citusdata / django-multitenant

Python/Django support for distributed multi-tenant databases like Postgres+Citus
MIT License
730 stars 118 forks source link

Add support for backwards migrations to migrations.Distribute #106

Closed JelteF closed 2 years ago

JelteF commented 2 years ago

Since Citus 9.5 you can now undistribute a distributed table. This allows us to easily add support for backwards migrations.

To test this we now run all of the migrations backwards (except for one). Most of the raw sql, I did not add a reverse migrations for. This seems unnecessary, because a migrations down the line will delete the whole table anyway.

Because we run these backwards migrations, this also serves as a regression test for the problem that PR #95 fixes. Inbetween test runs we now clean up the whole database and the migrations are run for each Django/Python version combination that we run. So if newer Django versions break the migrations we'll notice now.

maziar-dandc commented 2 years ago

can also refactor

if settings.USE_CITUS:

into Distribute class instead of it being required inside migration files, composite key creation can also be Classed and then you'll have stuff like:

citus_migrations.MakePKComposite('cms.Plugin'),
citus_migrations.Distribute('cms.Plugin'),
JelteF commented 2 years ago

The settings.USE_CITUS thing is only meant to be used in tests. It's not meant as an API exposed to users of the library. So I'll keep that as is.

maziar-dandc commented 2 years ago

@JelteF makes sense.

In our case, running tests in development on a citus db is too slow, so we opted to use a normal postgresql db for tests & use this variable for the difference. Cannot imagine test rebuilding the db and running distribute() command on 100s of tables, would be snorting by the time its done 😅