seppevs / migrate-mongo

A database migration tool for MongoDB in Node
MIT License
931 stars 166 forks source link

Orchestrate db migrations for 2 simultaneous starting pods in a k8s environment #206

Closed BorntraegerMarc closed 4 years ago

BorntraegerMarc commented 4 years ago

Is your feature request related to a problem? Please describe. I'm wondering whether this library takes care of a problem that we're currently having in production environments (it's a very common problem). What is happening:

  1. In our kubernetes production cluster we have of a Deployment 2 replica pods running (that means that our server runs 2 instances for load balancing).
  2. On every server startup we check if DB migrations need to be run
  3. Now, if we release a new version including a new DB migration the 2 pods would simultaneously start & run the exact same DB migration. Which is not good because DB migrations should only run once.

This is especially a problem for migrations that take multiple seconds to finish.

Describe the solution you'd like I honestly don't have a solution for this problem 🙂

Only thing I can say is that ideally this lib "knows" when a pod has already started a migration and doesn't start it again. Some of the "bigger" db migration tools take care of this.

Describe alternatives you've considered None :(

Additional context

seppevs commented 4 years ago

Sorry, but I'm going to close this issue. I don't think it's a problem migrate-mongo should solve.

205g0 commented 3 years ago

@BorntraegerMarc have you found a solution to your problem?

BorntraegerMarc commented 3 years ago

I definitely think this is a problem that the db migration tool should sove.

Since the owner said that migrate-mongo will not be solving this problem I'm relying on a different db migration tool: https://github.com/db-migrate/node-db-migrate/

In the next major version we'll get a feature State Manager: https://github.com/db-migrate/node-db-migrate/issues/538

@205g0 The other obvious solution is to always remove all pods except for 1 and then run the db migration in only that pod and then to start up all other pods again. Depending on your workload of the cluster this might not work for you though in production