search

db-migrate / node-db-migrate

Database migration framework for node
Other
2.32k stars 360 forks source link

Multiple migration directories? #756

Closed joezappie closed 2 years ago

joezappie commented 2 years ago

I'm submitting a...

Question/Request

I'm switching from postgres with Knex to mongodb and am looking for a migration tool. A nice feature that Knex has is the ability to specify multiple migration directories and I'm wondering if this can support my needs.

The reason for multiple migration directories is I have a bunch of "generic" migrations that I apply to all clients. I also need to be able to customize a DB for specific clients. In Knex, I had a general migration folder and then a folder for each client. I could then specify both folders, it would take and merge them together and run all of the migration files.

I see that this supports scoping, but I don't believe I can accomplish this with that? Could this scoping have a flag to run migrations above it?

For example:

Then possibly specifying an --recursive flag?

db-migrate up:client_1 --recursive

Which would load all of the migrations in the main and client_1 folder in the correct order:

  1. migrations/1633521000_users.js
  2. migrations/client_1/1633553000_users.js
  3. migrations/1633572000_sales.js

I do plan on using the API, so additionally it would be nice to have this in there as well. Maybe like this?

create(migrationName[, [scope], [recursive], [callback]])
up([[specification][count], [scope], [recursive], [callback]])
down([[specification][count], [scope], [recursive], [callback]])

Alternatively, maybe it would be easier to follow next and just allow the config to specify either a string or array for migration dirs (tho I don't see anywhere in the configs to specify migration dir except via the command line flag?). Snippet from Knex's documentation:

directory: a relative path to the directory containing the migration files. Can be an array of paths (default ./migrations)

Workaround

Maybe its already possible by just running each one separately but I'm not sure how this keeps track of its last migration. Im assuming if I ran db-migrate up and then db-migrate up:client_1, the client_1 migration wouldn't run because its timestamp is older than the last migration in the main folder?

wzrdtales commented 2 years ago

Concerning your problem

right now scopes can be executed alltogether with up:all. But that wont really help you in your described use case. You can run scopes one by one yourself of course as an immediate workaround, but effectively you want groups of scopes. Db-migrate does execute everything in alphabetical order right now, that would apply to up:all as well.

Your described --recursive functionality is btw. wrong. recursive would take subfolders in client_1 so this wouldn't solve your problem anyways. It is correct that your problem with intertwined migrations would remain in this case and this would need to mixin those migrations.

My view on the problem

I see groups problematic a little problematic, since you have to strictly maintain the dependencies correct (which you realized here as well). What I would consider a more appropriate would be a depends on configuration for scopes and migrations, which would enforce execution of those and optionally also executes them. This is indeed something I considered before already in the design process for some of the other newer (yet unofficial) parts of db-migrate.

I will generally consider this input for the future design since the execution order is being redesigned.

wzrdtales commented 2 years ago

Maybe its already possible by just running each one separately but I'm not sure how this keeps track of its last migration. Im assuming if I ran db-migrate up and then db-migrate up:client_1, the client_1 migration wouldn't run because its timestamp is older than the last migration in the main folder?

The scopes will run individually to answer that question also. So the timestamp from any other scope will not influence the others.

joezappie commented 2 years ago

@wzrdtales Thanks for the info. I'm using the API to make my own utility scripts so having the ability to run each scope individually is enough for my needs.