search

db-migrate / node-db-migrate

Database migration framework for node
Other
2.32k stars 360 forks source link

The new roadmap to v1.0.0 #627

Open wzrdtales opened 5 years ago

wzrdtales commented 5 years ago

Foreword

This year is special to me, since I am finally able to provide more of my time to open source. First of all due to the commitment https://github.com/db-migrate/node-db-migrate/issues/605 of sponsoring time and money from my own company (which currently means effectively plan in whole days/weeks of time that I do not spend on paid projects and invest those into open source instead). This is important, since nothing is free in this world and this is my new way, to give something back to the open source community, since a lot changed for me and I do not have as much free time as I wish I would have.

Further I added opencollective and sponsoring slots, from which funds also solely progress and work on db-migrate will be paid. So this way also people that are not able to contribute to the project in terms of providing PRs and their workpower, can contribute to the success and further development by providing resources otherwise.

That being said, i am very grateful for all the contributors of the past and of the future of any kind. All support matters!

The roadmap to v1.0.0

That being said, the new roadmap which we try to achieve over the next few month consists out of the following:

Work In Progress

The good news here are, a lot of this is done or in progress (master branch, not released yet). For example the structure has been reworked completely already. Yet I am thinking to also replace the CLI, but in favor of a shorter release time I might drop that into the next step. What is in its current state already ready is the new migration schema with a lot of new features. More to that later. The next step will be the templates and the default columns for table creation. After that I either go on the seeders or the concurrency manager, which highly depends on how the dependencies of this task resolve to each other (it turns out, most if not all tasks are somehow related to each other and require each other).

Also this is a very intensive rework and a lot of work put into thoughts and new processes, I am looking to keep the breaking changes to a minimum. In fact, I will never drop the version 1 migration schema. So expect that most should continue working, except that we're completely dropping support for old node versions, to move the db-migrate code base to a more modern state. This should be fine for the majority, for everyone else that does not integrate the db-migrate api, this change shouldn't affect to badly either. But it is finally time to move on and drop the support at this stage (also noting that node v6 is almost out of the maintenance cycle anyways.

The new migration schema

So the biggest thing in this release will be that I will finally clean up with the missing seeders and get the strings straight on this point and of course the migration schema. Which is leveling up the game a lot.

So to give you a preview of what is coming (and already working, if you're feeling fancy checkout the master branch and pg driver which already carry support). Here is what it is about:

The migration schema v2

The migration schema v2 is a very strict new standard, in which ultimately only reproducible components are allowed to be executed. It adds support for atomicity without transactions, it is to a certain degree self healing and it massively decreases the complexity of creating migrations. It allows an as seamless process of migrating up, as rolling back. It clearly defines guidelines for zero downtime compatible migrations and some more compelling features which will incorporate the DDL as well as the DML.

To give you a more alive example, lets take a series of actually working v2 migrations.

So assume this two migrations:

Migration 1

'use strict';

exports.migrate = async (db, opt) => {
  const type = opt.dbm.dataType;
  await db.createTable('test', {
    id: {
      type: 'UUID',
      primaryKey: true
    },
    test: {
      type: type.INTEGER
    },
    test2: {
      type: type.INTEGER,
      notNull: true
    }
  });
  return db.createTable('test2', {
    id: {
      type: 'UUID'
    },
    test_id: {
      type: 'UUID',
      foreignKey: {
        name: 'namename',
        table: 'test',
        mapping: 'id'
      }
    },
    test: {
      type: type.INTEGER
    }
  });
};

exports._meta = {
  version: 2
};

Migration 2

'use strict';

exports.migrate = async db => {
  await db.dropTable('test2');
  await db.removeColumn('test', 'test2');
};

exports._meta = {
  version: 2
};

Explanation

So what you can see here is, migration 1 creates two tables and migration 2 deletes one of the table completely and and removes a column from the other one. The first thing you will notice is that the interface looks different. First of all there is no up and down anymore. Only a migrate and a required _meta export are provided. The latter will later also allow to introduce some more features like specific driver instances per migration later on. The second change in the interface is, that callbacks are dead. Instead the migrate function now gets passed the driver and an options object. The options object contains the objects you used to retrieve through the setup function and might also hold some other toolings in the future.

So all in all the migrations look and feel very simple now and you might ask, where is my definition of rolling back? The answer is, there is none and you don't need any. The reason for this is the same reason why the new standard is so strict and the same reason why the v1 standard will keep alive forever. Certainly everything can be covered in the v2 schema, but in the meantime it will always be possible to execute operations that are not yet executable on the v2 schema. But that aside. Why you don't need the down operation anymore?

db-migrate got a few new components exclusively available to the v2 schema (with some exceptions such as manual schema feeding which will be available for inter compatibility to the v1 as well). These components are for example a component that learns about the manipulation of the DDL you're applying currently and another that stores the state of the new manipulated schema and information about the alteration that has been made. When you now execute an action in the new schema you're not accessing the driver directly anymore. Instead db-migrate will inject a chain of actions before it sends your operation finally off to the driver beneath. This allows db-migrate not only to intercept but also to correct actions. For example the new migration v2 schema also comes with failureStrategies. Currently the only available and also the defacto standard is rollback the current migration on failure automatically. Should the migration being interrupted by for example a failure of the database in the process of migrating, then db-migrate will be able to recognize this on the next run and rollback the executed actions before continuing by default. The rollback is being recreated from the stored information. That also means, when you decide to delete a table for example, db-migrate knows exactly how your table looked before it was deleted and can completely automatically revert to the before state. An exception to this, is of course the data in the table. This is why a dropTable option is not safe by the zero downtime guidelines and a new deprecateX function for tables and columns will be added to the driver. Those will support a full zero downtime compatible cycle. Which will consist out of the following step:

These deprecation functions will also get later on support for signaling from the application directly, to shorten the process and they require information about a new release. For this there will be _meta export signal to define a certain migration as a start for a new release. This also might be improved in the future, since there are plans to also implement new migration patterns, to organize migrations in features and versions directly (for example a long requested feature like semver versioning might become reality, but this is yet to be defined).

Some final words about the standard itself. There will be a definition of the standard, which might be extended with features, as long as it is not altered. So already defined components might only change in a new version of the schema, which shall be avoided when possible. But new components might be added to make it more feature rich.

Some more notes

As always I want to express, I welcome any help. Be it documentation, implementation or helping out others with their issues. I want to thank all the people that helped in the process and all the contributors that added value to the project and I am looking forward to make db-migrate a migration framework that is not staying in your way, but helping you to do the right things and get the things done. Especially important to me is to search for a way to transfer knowledge, especially on the zero downtime section. I don't know yet, how I best accomplish this, but I consider this being even more important than everything else since I might provide a powerful tool, but what does it help you if you don't know how to handle it. Whoever has ideas or wants to help out on that section. I greatly appreciate your input!

Post v1.0.0

So to answer the question, what comes post v1.0.0. I can't answer yet, but there will always be room for improvement and i want to encourage you the community, to take active part in creating meaningful value for the project.

Notable references

Issues to be checked further

--- Want to back this issue? **[Post a bounty on it!](https://www.bountysource.com/issues/74943541-the-new-roadmap-to-v1-0-0?utm_campaign=plugin&utm_content=tracker%2F73887&utm_medium=issues&utm_source=github)** We accept bounties via [Bountysource](https://www.bountysource.com/?utm_campaign=plugin&utm_content=tracker%2F73887&utm_medium=issues&utm_source=github).
wzrdtales commented 5 years ago

/cc @BorntraegerMarc

wzrdtales commented 5 years ago

To add some afternotes, v1 migrations will also benefit from concurrency control, should the driver beneath support it, but they won't be able to automatically cooperate with v2 schemas. For that manual feeding of the schema will be added, so that it is still possible, to run rawSql and apply the alterations made manually.

stale[bot] commented 5 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

wzrdtales commented 4 years ago

some significant progress lately.

next up is

BorntraegerMarc commented 4 years ago

@wzrdtales would it make sense to release a release candidate? Of course only if you think API is not gonna change :) I would be glad to test a RC in my biggest projects as well.

wzrdtales commented 4 years ago

there is a beta tag released already :)

BorntraegerMarc commented 4 years ago

there is a beta tag released already :)

yeah, saw that :) I was specifically referring to a release candidate because I would like to wait with adoption until we're sure that the API is not gonna change anymore. I can invest some time to test pre-releases but don't have time unfortunately to re-implement changes, when they occur...

wzrdtales commented 4 years ago

I will make sure to hit you up as a test candidate as soon as I freeze the standard (actually lot of it is alreay freezed).

which drivers are you using btw.? currently I implemented the support for the new functions only on pg and cockroachdb.

BorntraegerMarc commented 4 years ago

which drivers are you using btw.?

I'm using MongoDB

wzrdtales commented 4 years ago

Also decided to remove the transitioner entirely. It was a utility to help migrate away from the very old async libs and globals from the first incubation before I rerolled the package completely. Will be available in the older versions should anyone need it to still migrate away pre 0.9.x migrations.

Note to myself: add to breaking change list.

wzrdtales commented 4 years ago

tunnel ssh will move into the plugins, again

Note to myself: add to breaking change list.

dimitriy-tomilovskiy commented 4 years ago

@wzrdtales Any news on concurrency manager? It could be mighty useful for some of the things we are looking to do.

levino commented 3 years ago

Where are the docs for the beta versions? I was unable to find them, sorry...

rysolv-bot commented 3 years ago

themanmaran has contributed $75.00 to this issue on Rysolv.

The total bounty is now $75.00. Solve this issue on Rysolv to earn this bounty.

nilligan commented 3 years ago

Would it be possible to set a different, non-latest tag to the beta releases? The current latest tagged release is for 1.0.0-beta.14, which means that anyone running npm install db-migrate installs a version of the library that is still in beta (rather than latest stable 0.11.11)

From the npm-dist-tag docs: https://docs.npmjs.com/cli/v6/commands/npm-dist-tag

Publishing a package sets the latest tag to the published version unless the --tag option is used. For example, npm publish --tag=beta. By default, npm install (without any @ or @ specifier) installs the latest tag.

It also produces (what I assume is) unwanted behavior from the npm outdated command

> npm outdated
Package                    Current   Wanted         Latest  Location
...
db-migrate                 0.11.11  0.11.11  1.0.0-beta.14  <redacted>
...
wzrdtales commented 3 years ago

ahh i hate npm for being inconsistent... It was indeed not wanted, I already wondered why suddenly more people are starting to use the beta. A tag created with the prerelease flag really should just being treated as such without the stupid --tag flag... (it was previously published as thus, but I didn't use that flag a long time and just forgot about its existence again assuming npm was designed intelligently)

I will see to push a "new" old version out, however, the latest version is pretty stable, it is in use in huge production systems (however many of them maintained by me where I am dogfeeding the new migration schemas and by that battle testing them before the release) and there are really not much issues, especially if we leave out the new functionality.

nilligan commented 3 years ago

Totally agree on wishing that npm treated the latest tag differently...

Would be very much appreciated if you could publish a 0.11.12 and add a different --tag option to future beta releases

And thank you 🙏 for all your work on testing v1.0.0!

mriedem commented 2 years ago

Hi, I'm just wondering about any kind of status updates on the 1.0.0 effort. We're probably going to upgrade from 0.11.12 to the latest beta (currently 1.0.0-beta.18) because we're getting vulnerability scan hits on old transitive dependencies. The beta releases have been happening for 3 years now and the last beta release was 5 months ago. I'm assuming the beta versions are pretty stable (given they were considered stable in 2020 per the common above), I'm just wondering if there is an expectation of when the beta tag is dropped and 1.0.0 becomes official?

wzrdtales commented 2 years ago

v1 is for long time in use in our projects where we battle test it, there are still some issues with the v2 migrations although it is very stable by now, if you rely on the v1 schema but use v1-beta of db-migrate it will be rock solid though. We have been scaling for the last period a lot company wise, so that ate a lot of time, we still are involved in db-migrate and continue to support it. It might be really time to consider dropping out v1 without the final support of the v2 schema, I would need to check if there was any breaking change pending to avoid another major version jump.

wzrdtales commented 2 years ago

Also worth mentioning, we are the biggest investors as a company behind db-migrate right now, but we support more than just db-migrate as a open source project, so it is not like we could expand the efforts we already make (also not everything is visible directly to everyone, especially considering the v2 efforts and driver related things like the stuff on the cockroachdb driver). So obviously it would help the project to get more backers and funding, and individual contributors to accelerate the pace.