Migrations can't run if following INSTALL.md

[frankduncan]

INSTALL.md has the line to run node data/import_into_postgres.js When run against an empty database, which would be the case if you are starting from ground zero, this will generate all of the tables specified by the models in your ./models directory via sequelize. If you have been developing, and generating migrations for model updates, then someone starting after you've merged your work in, upon running sequelize db:migrate will see failures as their database structure already incorporates the changes that for which migrations were created.

This should be changed such that there's a core sql file that is locked to a certain development version of the data model, then all changes moving forward are migrations built on top of that structural file.

[kfogel]

I think I understand how this isn't this the situation we talked about yesterday on the phone, but let me check:

The problem is essentially that for new developers, we documented have a data import step that produces tables that don't match what's currently in production, because what's in production includes a set of manually-done "migrations". IOW, data/import_into_postgres.js is out of date: it was never updated to include DB schema changes that are actually live in production. This doesn't matter for production, because we'll (in theory) never run data/import_into_postgres.js there again. But it does matter for deploying a staging server, because now we have no way to stand up the sample data with the latest code.

If so, then shame on me for not noticing, in code review of those DB-affecting changes, that data/import_into_postgres.js was increasingly out of date. Your solution makes sense, and is essentially what we talked about yesterday, it's just that when we talked we either forgot about data/import_into_postgres.js or we assumed it was up to date, and so we only talked about the older out-of-date migrations and how they wouldn't be needed anymore.

[frankduncan]

Actually, the opposite turned out to be true.

The normal development workflow is: Have development database, Update model, create migration the mirrors model update in database, push to production, run migration at same time as doing a git pull

But! The way the installation instructions work is that if you are a new developer and you come in with no development database but are working on the code with the changes in the model already applied, you generate the development database FROM the current model (this is a feature of sequelize). Then, the migrations that were written to go hand in hand with model changes are now invalid for your database. This means that, in reality, if you're coming into the project after a year of migrations have been added, those may all fail on you and you have to work it out to get your migrations and the current model to all be in sync.

This seems like a design weirdness on the part of sequelize, so it's probably that out codebase isn't using it properly or something? I don't know.

What I do know is a fix is to create a snapshot of the database at a given timestamp, and say that you just load from the sql file, and then run migrations, and stop letting sequelize play with your database.

[frankduncan]

I should note that this is an orthogonal problem to what was going on with staging/production. But I ran into this while trying to create a good fix/snapshot for how we should do migrations in staging/production, so it should solve both at the same time.