deconst / strider-deconst-control

Deconst control repository build plugin for Strider CD
MIT License
1 stars 2 forks source link

Review build config & troubleshooting processes #31

Closed meker12 closed 8 years ago

meker12 commented 8 years ago

Recently, had some issues with refactoring content for vmWare docs that led to a number of other issues. Here are some questions that came up while trying to sort out the problem:

@kallimachos, @bhpurswell @klevenstein

smashwilson commented 8 years ago

Alright, I'll give these a shot :smile:

What causes github webhook failures for Strider build, and how do you resolve them?

I've seen a few webhook delivery failures myself and I have yet to track it down. They're usually 502s, which means that either nginx or the cloud load balancer are rejecting the request because an upstream service is unavailable, but nothing seems to be in the logs.

In the short term, as long as Strider still looks like it's up (meaning you can visit it with your browser), you can re-deliver the webhook payload from the "settings" page of your repository. That should kick off the build that was missed.

Does the build environment cache content that might make the PR build output incorrectly? If so, is there a way to clear the cache?

No. Strider does offer a build cache, but our build plugins don't use it. The content build plugin explicitly wipes the envelope and asset output directories before it begins, too.

Confirm that the requirements (best practices) for content-id base values are documented (path length, escape character list or how to avoid escaping, trailing slash, and so on)

From https://deconst.horse/writing-docs/author/#adding-a-new-content-repository I have:

"The most important setting within this file is the content ID base. The content ID base will be used to uniquely identify the content produced from this directory within the system, so it must be unique across all content repositories that are published to a Deconst cluster. The easiest way to accomplish this is to set the content ID base to the content repository's GitHub URL (including the trailing slash to be consistent)."

That's a partial answer. Beyond that, it's helpful to ensure that each content ID base is a valid URI; there are already certain instances where we parse them as such. Somewhere on the backlog is to enforce that more strictly (deconst/content-service#75), and even further on the backlog is changing them to the deconst:// scheme described in deconst/deconst-docs#237.

The authoritative source for escaped characters and so forth would be RFC3986 which describes the actual grammar, but it's a bit dense. There's a StackOverflow question that summarizes it as "only alphanumerics, the special characters $-_.+!*'(),, and reserved characters used for their reserved purposes may be used unencoded within a URL," with some updates for the characters valid in the hostname component.

As for maximum length: keeping with the valid-URL theme, it looks like 2000 characters is a practical maximum. In actuality, because the final URL's path uses only a suffix of the content ID, you could likely get away with packing hundreds of megabytes in one, as long as it was mapped with hundreds of megabytes in the control repository, too. Please don't do that, though.

Trailing slash is purely a matter of personal preference and consistency. The preparers and the presenter all normalize it anyway. I try to include the trailing slash everywhere but there are plenty of places I've forgotten.

What happens when you have content mapped to staging.horse, and then you remove it and map it to a production repo? Does the build job get recreated?

Preview links created while the staging.horse configuration was active will begin to 404.

Nothing will happen or needs to happen with the build job; content mappings are designed to be changeable independently of content builds. You can ship new content to an existing mapping or change a mapping to existing content without changing the other. The idea is to let different people change sitewide IA and navigation and such without having to worry about the content itself needing to change.

As long as the content IDs remain stable, the mapping can change at any time.

Installing local deconst environment for testing -- Will deconst client still work?

No, it's been broken for some time now. I shifted effort entirely to make the pull request builder work instead, because it serves a larger audience.

You could use deconst/integrated with Docker and Docker Compose to do local builds. I use it actively for integration testing so it is maintained. It does make you manually run script/add-assets, script/add-sphinx, and script/add-jekyll to submit content, though.

I noticed this morning that the scripts also doesn't play nicely with the Docker for Mac beta. I'll probably fuss with them to support both that and docker-machine at some point.

How-to question about search results and how long it takes for new content to show up.

The how-to repository uses a Google custom search, not Deconst's search API, so I don't have any insight into that one. Last I heard, we were trying to track down the appropriate Google credentials, and considering replacing it with a Deconst-backed search (rackerlabs/nexus-control#396).

smashwilson commented 8 years ago

@meker12 Did this answer your questions adequately? I know it's a wall of text, I was trying to err on the side of completeness.

meker12 commented 8 years ago

Thanks for the information. A few clarifications:

Using the staging.horse for development builds

Adding a mapping file for staging.horse enables PR builder support for new content.

For content deployed to staging.horse, is the content staged in a different content store than the one used for the production builds?

When we remove the mapping from the staging.horse.site, do we need to do anything to delete content from the content repository? (Think not since you said that you will get 404s after removing the mapping.)

Note: We have a dev team that will be developing docs for a beta, and I want to configure their project to use the development branch so they have preview capabilities, but content isn't deployed to production.

Deconst integrated

So it is possible for me to install this locally, and probably a good idea so we don't have to preview nexus-control updates on the production site. The Docker Beta issue for Mac isn't a show stopper?

Content API When we remove content the site or remap to a different URL, we need to get rid of the old content so it's not indexed for search. Can people on our team use the content API to delete content from the content store? Do we need special permission? I'm assuming that the content API operations are documented.

Strider Can you give someone on our team the appropriate permissions on Strider so that we can add people as needed.

smashwilson commented 8 years ago

Alrighty, next batch:

For content deployed to staging.horse, is the content staged in a different content store than the one used for the production builds?

Yes. Envelopes created for pull request builds (for any domain) is put in a staging database within Mongo, not the production one. You can't access staging envelopes from the production environment.

When we remove the mapping from the staging.horse.site, do we need to do anything to delete content from the content repository? (Think not since you said that you will get 404s after removing the mapping.)

Correct, you don't need to do that. It won't be accessible and the storage is negligible.

So it is possible for me to install this locally, and probably a good idea so we don't have to preview nexus-control updates on the production site. The Docker Beta issue for Mac isn't a show stopper?

I worked through the Docker for Mac issues last week, so it should be working correctly now; I was using it with Docker for Mac this morning. It should still work with Docker Toolbox as well.

I'll add setup instructions into nexus-control's README.

When we remove content the site or remap to a different URL, we need to get rid of the old content so it's not indexed for search. Can people on our team use the content API to delete content from the content store?

There are two cases here.

Deleting content within a content repository -- removing a single page within a Jekyll or Sphinx repository, but keeping the rest of the content around -- will be handled correctly already. When the deletion is merged into master, the content will be deleted from the content service.

Deleting an entire content repository or changing the content ID base of published content, which are both the same thing from Deconst's perspective, will not be handled correctly. The old content IDs will need to be deleted through the API.

Also, remapping content to a different URL doesn't require any deletion. Remapping content in the control repository changes the mapping of URLs to content, but the content itself doesn't change. The URL doesn't give a specific piece of content its identity; a content ID gives a piece of content its identity.

Do we need special permission?

You'll need an API key. You should already have at least one somewhere in PasswordSafe. If anyone else needs one, I can provide it.

Er, it also won't work on VPN, because port 9000 is blocked and the content service endpoint is hosted at https://developer.rackspace.com:9000/.

I'm assuming that the content API operations are documented.

They are indeed, in the content service's README, including the content deletion endpoint.

Can you give someone on our team the appropriate permissions on Strider so that we can add people as needed.

You or @klevenstein can add or remove people from the "nexus-build-access" team in rackerlabs to grant or revoke access.

When you add someone new, wait for them to authenticate the first time at https://build.developer.rackspace.com/, then re-run the rackerlabs/nexus-control build within Strider to grant them access to all of the existing Strider builds.

smashwilson commented 8 years ago

Closing this because I think I got all of your questions. Re-open if anything here is unclear or open new issues as you have more questions!

👍