storage/sources: in-place schema changes

[nmeagan11]

Product Outcome

Backwards compatible schema changes to upstream systems can be incorporated into Materialize without requiring users to drop and recreate their (sub)sources. A backwards compatible schema change is the addition of a new column with a default, or deleting an optional (slash nullable) column.

We should think about how this new feature should interact with the current workflow for blue/green deployments (i.e. how to redeploy the minimum viable dependency graph with no downtime).

Related issues:

• [ ] (Pre-requisite) https://github.com/MaterializeInc/materialize/issues/20208
• [ ] (Pre-requisite) https://github.com/MaterializeInc/materialize/issues/24843

Discovery (archive)

• Reusable recklocking
  • Recent idea from @benesch: https://github.com/MaterializeInc/materialize/issues/20208#issuecomment-1955401921
  • https://github.com/MaterializeInc/materialize/issues/17021
• https://www.notion.so/materialize/Storage-Roadmap-Brain-Dumps-4427323cbb54498f9950b6c466eec366#30cfca2c32174ef5adb97a3328c50387
  • Initial thoughts (early 2023)
• https://github.com/MaterializeInc/materialize/issues/22836
  • Describes basic support for backwards-compatible schema changes for tables.
• https://materializeinc.slack.com/archives/C01CFKM1QRF/p1706661692662929
  • Slack conversation about allowing multiple subsources referring to the same upstream table.
  • https://github.com/MaterializeInc/materialize/issues/24843
• https://materializeinc.slack.com/archives/C05SLRXKNGN/p1706729350801729
  • Slack conversation about perils of today's blue/green approach when used with sources

Decision log

• 31 January 2024. Move to "Later"/"Discovery" on the storage team roadmap, with no release date.
• 31 January 2024. Convert to new epic format.

[sploiselle]

Just for some context, doing this likely requires #16650 or a stable interface between HirScalarExpr and the dataflow layer-- in addition to much other work.

[benesch]

It's maybe a little surprising given the issue title, but https://github.com/MaterializeInc/materialize/issues/17021 contains a proposal for how ALTER SINK could work.