dipdup-io / dipdup

Modular framework for creating selective indexers and featureful backends for dapps
https://dipdup.io
MIT License
97 stars 52 forks source link

Use `aerich` for database migrations #1110

Closed bigherc18 closed 1 month ago

bigherc18 commented 1 month ago

This PR is about using aerich to add support for database migrations

Here's a demo: https://asciinema.org/a/Sw7OwQZKmWyznV3xgohjd59OT

I added a new config called migrations_dir to specify where migration files should be stored

database:
  migrations_dir: ./migrations

A a set of commands have been added to manage database migrations:

Theses commands invoke aerich's commands under the hood

The migrations directory should be initialized before using any of these commands, either with dipdup schema init or by running the indexer with dipdup run.

Note that aerich commands can't be used directly, as it expects the config to be stored in pyproject.toml

bigherc18 commented 1 month ago

Do you guys prefer rebase or merge with next to fix the conflicts ?

droserasprout commented 1 month ago

@bigherc18 We squash every merge request, so it's up to you what's inside.

bigherc18 commented 1 month ago

Some concerns and limitations I have:

bigherc18 commented 1 month ago

I think, there's nothing critical. Yes, running commands in random order on multiple databases sometimes confuse aerich, but all real cases works perfectly.

Thank you for your feedback but I just want to make sure I explained my concerns clear enough for you

  1. Multiple databases not supported by migrations

TLDR; if a user uses multiple databases at the same time, migrations is not supported

I see dipdup makes it easy for users to swap databases by just combining different config files with multiple -c args, the problem is: if a user swaps his database after doing some migrations in the first one, aerich is now really confused and I don't even know what will happen, the aerich table will be empty (or contains migrations history for the second database) but the migrations folder still contains files from the first database

I think this worth at least a note in the docs

  1. CLI handing

Here's a little video demonstrating the issue:

demo