Phil/cpv2

[psFried]

Description:

Rolls up whole bunch of work from both @jgraettinger and myself related to:

• publications
  • builds
  • validation
  • activation
• inferred schema handling
• materialization schema evolution (onIncompatibleSchemaChange)
• controllers :sparkles:

I recommend reviewing this commit by commit, and tried to write helpful commit messages. Here's a high-level summary:

Reframe validation to be an operation over a tables::LiveCatalog (a snapshot from the control plane) and a tables::DraftCatalog, which is able to represent deletions.

Updates the publications handler to:

• work with the new validation crate changes
• use optimistic locking instead of holding row-level locks
• setup for a future publish function that can be invoked by controllers
• use a more minimal spec expansion
• trigger controller runs after a successful publication

Introduces controllers, which are background automations that are tied to specific live specs. Controllers are responsible for:

• updating shard and journal specs in the data plane, in response to a publication
• keeping specs up-to-date with respect to their dependencies
• updating collection inferred schemas (inferred schemas are now added directly to the collection spec)
• updating materailizations in response to incompatible schema changes
• updated materializations in response to changes of their sourceCapture
• maintaining a status that can provide some basic information about their operation and any errors that occur

Workflow steps:

The control plane should work more or less as it has in terms of user-facing operations, with a few exceptions:

• You can now delete and re-create collections and tasks with the same name :tada:
• Inferred schemas now work with derivations
• Inferred schemas now get baked into the collection model. This means that you can technically now overwrite the published inferred schema with flowctl. This might be easy to do, since it's hard to know if the inferred schema has changed, and your local spec is now stale.
• Materializations now obey the onIncompatibleSchemaChange property (top-level, or can be overridden on a per-binding basis)
• Users will need to update flowctl in order for things to work with the new control plane. One reason for this is that the spec_type column on live|draft_specs is now non-nullable.
• Activations now happen asynchronously, after the publication itself has already succeeded. Thus there is not a possibility of a slight delay after publication before the new shard status can be shown on the "details" page. This means a user can publish a task that was failing, and then get redirected to the details page and possibly be confused by the shard status still showing as failed. One way to address this might be to have the details page query shard status a little more frequently. We also have the information available in controller_jobs.status->activation->last_activated, which can be compared agains live_specs.last_pub_id to see if the activation has been completed yet.
• We no longer return JobStatus::TestsFailed when a test run fails. Instead it just returns BuildFailed (the can be fixed but I wasn't sure if it was important).

Documentation links affected:

(list any documentation links that you created, or existing ones that you've identified as needing updates, along with a brief description)

Notes for reviewers:

I'd appreciate any time you have to test this yourself locally, and report any issues. I'd like to continue testing, myself, particularly on the transition from the old version, and I expect I'll need to push some updates before this is ready to deploy. I'd like to keep this PR open until we feel this is in a deployable state.

Please reach out by any means with any questions.

Deployment plan

The basic steps for deploying this are the same as any other control plane PR involving a database migration, except that there's an additional migration to be run after the deployment finishes. This second migration will actually need to be run multiple times, because it will incrementally enable controllers for pre-existing live specs, in batches of 1000 at a time. This gives us a chance to bail out if things aren't working right, instead of just unleashing controllers on all of our precious live specs. The end-to-end high level plan is thus:

• Run the 50_controllers.sql migration, previous agent versions will continue to run against the new db schema
• Deploy new agents, just using the normal ops repo
• Verify that we can publish a new task and edit an existing one through the UI, verify controller runs in controller_jobs
• Run the 51_backfill_controllers.sql migration, and spot check the controller runs to make sure things look good
• Repeat the previous step until we've backfilled all of them
• Profit

---

This change is 

[jgraettinger]

This means that you can technically now overwrite the published inferred schema with flowctl. This might be easy to do, since it's hard to know if the inferred schema has changed, and your local spec is now stale.

flowctl now adds expectPubId to any live specs it pulls down from the control plane. expectPubId of zero means "must not exist".

This is to prevent a user from inadvertently overwritting an updated live spec by publishing a stale spec from their checkout. They'll get an error because expectPubId will disagree.

They'd have to explicitly remove the expectPubId field to denote that they want their local version to take precedence.

flowctl tries not to publish a spec which isn't actually changing, so they should be good to leave unmodified specs as-in in a local checkout while publishing a managed spec over-and-over.