In-place major version upgrades

[msullivan]

Currently, while we allow in-place upgrades between minor versions, upgrading between major versions requires a dump and restore. This is slow, and requires downtime.

Currently we use the patch system to make catalog changes during minor version upgrades, but using the patch system for major versions is probably unworkable:

1. The number of patches needed would be much higher, which is problematic because they are a pain to develop and maintain. Currently we only need patches when a backported bugfix requires a catalog change. Now it would be needed for all features on the main branch.
2. Worse, there is no good way with that approach to populate new fields in the reflection schema.

The plan then, is this:

• edgedbstd gets a version suffix, so edgedbstd_vX
• edgedbstd still exists, and will contain trampoline functions and views that point to the equivalent functions/tables in the namespaced schema
• When compiling normal queries, the compiler can use the versioned std namespace directly, but when compiling user-defined functions, constraints, etc, it will use trampoline schema edgedbstd.
• When we do an upgrade, we will populate the new edgedbstd_vY schema, and then update everything in edgedbstd to point to it. (This probably means we can't ever remove fields from user visible tables, but that's probably OK.)
• The reflection schema also lives in the new edgedbstd_vX database. Before the upgrade, we will DESCRIBE SCHEMA the old database, along with all the schema object ids, (like dump does), and during the upgrade we will run that DDL script in order to populate the new reflection schema. (But we will not compile it to postgres, since the postgres tables already exist and are supposed to be compatible)

Eventually we might need to actually change layout, in which case some major versions might not be upgradable. But also there's a good chance we could do it gradually, with some tables or functions or whatnot using the new layout and allowing gradual backfill migration.

Open questions (partial list):

• This may require us to change our policy on postgres version support and new features. For example, we dropped support for postgres 13 in edgedb 4.0 because it didn't support multiranges. In-place upgrades increase the benefit of maintaining support for old postgres versions for longer. Maybe this would have made it worthwhile to disable multirange support on postgres 13, or to require it live in an extension.
• Maybe we could query pg_catalog to make sure that the database looks like we expect it to still?
• How do we handle extensions?

[msullivan]

I've focused mostly on the compiler component, but there is also a lot of room for cleverness to make this truly in place. With appropriate setup, almost all of this can happen while the previous version is still running, and even serving write queries. (As long as no migrations are being done)

[msullivan]

Other design notes (will edit):

• I think that scalar types in metaschema and the standard library still need to go into global namespaces, so we'll need to be careful there.
• Type view trampolines might be tricky, since (internal) schema fields can be dropped between versions. Do we need those really? We might need to dynamically generate the the trampoline switch by introspecting the existing views, and the new views, and generating something compatible with both.
• The compiler will need to be able to reference namespaced standard library tables, at least
• Do we namespace constraint trigger functions for the standard library? I think yes, and we don't trampoline them.
• Endpoint triggers might be tricky. We'll need to regenerate them if a user type points to schema object that has grown a new subtype.
• Extension upgrades will be tricky. We'll need to start properly versioning and having upgrade scripts.
• Extension configs need handling for views and configspecs

[msullivan]

Remaining TODOs:

• [ ] Extension upgrade support (https://github.com/edgedb/edgedb/issues/7724)
• [ ] Parallelize upgrade (by using compiler pool)?
• [x] Support upgrade while old version is running
• [ ] Block DDL while performing upgrade
• [ ] Better fault tolerance for finalize
• [ ] Support cleanup of prepared upgrade
• [x] Extension triggers and user functions: regenerate them if a user type points to schema object that has grown a new subtype