riverqueue / river

Fast and reliable background jobs in Go
https://riverqueue.com
Mozilla Public License 2.0
3.43k stars 90 forks source link

Push migrations down into drivers + alternate migration lines #426

Closed brandur closed 3 months ago

brandur commented 3 months ago

Here, push migrations down into individual drivers. Just like drivers define how the database they implement is queried, they'll also define how their database's schema should be defined.

The most obvious use of this is to support alternative databases. For example, given an SQLite driver, a lot of the DDL/SQL would be shared between Postgres and SQLite given they have a similar feature set, it won't be exactly the same, so both database drivers will need full control over over everything.

We also implement alternative migration lines. Each driver specifies at minimum a main migration line that acts as default, but can now provide an alternative line for its own purposes.

The two changes combined introduce another use: driver wrapping. A custom driver could wrap one of the main drivers, proxy through its main database migration line, but then augment that with its own alternative line as well.

Note that although I think this is in a position where it's mergeable, it's not quite complete in that we don't yet have a way of accounting for alternate lines in the river_migration table. Doing so will require the addition of a new field, which unfortunately requires a new River migration. I think there's a way we can work around that problem by augment river_migration from an alternative migration line for the time being, but TBD. It'd have the major downside of making migration code quite a bit more complicated because it'd have to operate against two possible river_migration schemas.

One aspect that might be slightly objectionable in this approach is that we do end up duplicating the full Postgres migration line in both riverpgxv5 and riverdatabasesql. I don't think there's a way around that as long as we want to continue go:embeding migrations. You can't embed from a directory above the current package, and symbolic links aren't supported by design. I've added Makefile and CI checks that provide easy ways of copying changes and verifying consistency, and with those I expect that this won't be too painful since we don't add new migrations very often.

brandur commented 3 months ago

Awesome. I'll get started on that next. Thanks!