Multi-phase preparer workflow

[smashwilson]

I've been talking about doing this for a while, but I haven't actually documented the full idea anywhere yet. This is what I want to do with the way that preparers work:

1. A preparer container (preparer-jekyll, preparer-sphinx) mounts the workspace into a volume. It's responsible for processing its input directory (CONTENT_ROOT or /usr/content-repo), writing each envelope to a url-encoded-content-id.json file in an ENVELOPE_DIR, and copying each asset to an ASSET_DIR.
2. A submitter container mounts the volume. It's also provided with the CONTENT_STORE_URL and CONTENT_STORE_APIKEY. It submits all of the assets contained in ASSET_DIR to the content store, then submits all of the envelopes from ENVELOPE_DIR. For bonus points and mad performance, it should do this in two HTTP transactions.

This lets us:

• Not duplicate and maintain the content store submission protocol across N preparers; instead, each preparer can write to disk, which is closer to what the native engines do, anyway.
• Build small, reusable audit containers that can (mostly) be used independently of the preparer.
• Further isolate the exposure of the content store API key - the submitter is trusted deconst code and never loads user-submitted content at all. The preparer (which often does things like exec the conf.py file) no longer needs an API key at all.
• Have much faster builds (I hope) by consolidating network transport into two HTTP transactions rather than a linear sequence of one per envelope and asset.

Here's my rough checklist:

• [x] content-service | Add API endpoints to the content store to accept batch envelope uploads as a tarball. (#133)
• [x] content-service | Bulk asset uploads as a tarball. (deconst/content-service#95)
• [x] content-service | Fingerprint and query fingerprints for uploaded envelopes and assets. (deconst/content-service#92)
• [x] preparer-sphinx | Modify the Sphinx preparer to follow the new preparer protocol (in such a way that won't break existing builds!).
• [x] preparer-jekyll | Modify the Jekyll preparer to follow the new preparer protocol (in such a way that won't break existing builds!).
• [x] submitter | Create the submitter and package it as a container. Use the new environment variable naming conventions from #101.
• [x] strider-deconst-content | Rework the preparation workflow to run the new sequence of containers rather than a single preparer container.
• [x] deconst-docs | :memo: the new build workflow and preparer container protocol. In the writer's section, cover the sequence of steps to rename content and put redirects in place.
• [x] :shipit: to deconst.horse
• [x] :shipit: to Nexus
• [x] integrated | Update the preparer scripts to run the new preparer flow.
• [x] preparer-sphinx | :hocho: cruft left over from before.
• [x] preparer-jekyll | :hocho: cruft left over from before.

Follow-on issues:

• [x] We're still submitting unnecessary envelopes and assets. deconst/submitter#11
• [x] The Strider build plugin doesn't handle the submitter's exit status. deconst/strider-deconst-content#26
• [x] Prefer assets from upstream in staging mode. deconst/content-service#103

[kenperkins]

It sounds like you're not planning on addressing (at least through this issue) any kind of differential asset upload. Is that correct?

[smashwilson]

Not initially, but this'll get us closer. Once we have bulk upload for envelopes and assets, we can add a handshaking request, where the submitter offers a set of checksums to see what it can leave out of the uploads.

[smashwilson]

I'm trying to keep this issue from becoming even more sprawling.

[smashwilson]

So, uh, now I am doing asset and envelope fingerprinting as part of this after all. Sprawl++

[etoews]

Is there a use case for eTags here?

[smashwilson]

Is there a use case for eTags here?

I don't think so, because our upload requests are performed in bulk. Part of my agenda is to accomplish a content repository publish with as few transactions as possible:

1. The submitter has a full set of assets and envelopes on disk. It fingerprints them and queries the content store API to ask what's new all at once.
2. The content store compares the fingerprints to the fingerprints of the latest resources. It returns asset URLs for assets that are already present and yes-or-no responses for envelopes.
3. The submitter prepares a tarball containing all new assets and uploads it to /bulkassets. The response contains the new asset URLs for those assets.
4. The submitter injects those URLs into the metadata envelopes.
5. The submitter prepares a tarball containing all new envelopes and uploads it to /bulkcontent.

ETags are more of a request-by-request sort of thing, and even if you could attach more than one to a single request, they wouldn't prevent the submitter from needing to prepare and POST this giant tarball anyway. Unless I'm misremembering how they work, of course.

[etoews]

Still had my head in the individual content envelope model so I was thinking request-by-request. nm

[smashwilson]

I've split the doctest work into its own issue at deconst/strider-deconst-content#25, because I'm close to shipping bulk differential uploads without it. It'll still be a pretty natural extension.

[smashwilson]

Here's a full build of the how-to repository, even with buggy duplicate asset and envelope detection:

🐎 🐎 🐎

[smashwilson]

:metal: This is now live and working.