Let's talk about migrations...

[bdunne]

Scenario: You have an app running. You create some new migrations and change code to depend on those migrations. During the new deploy, migrations fail. What should happen?

Migration 1: add a new column (unlikely to fail)

• Failure: the old pod is still in a good state and can successfully handle traffic. If the new pod were brought up at this point, it would probably be bad. (errors from missing columns)
• Success, the old pod is still in a good state and can successfully handle traffic. If the new pod were brought up at this point, its responses would probably be at least somewhat incorrect (new column has no data.

Migration 2: move data from an old column into the new column (most likely to fail)

• Failure: Transaction rollback, same state as Migration 1 success
• Success: the old pod is now in a somewhat bad state (old column has no data), new pod would probably be okay now.

Migration 3: delete the old column (unlikely to fail)

• Failure: Transaction rollback, same state as Migration 2 success
• Success: Old pod is now in a bad state (missing column), new pod is in good state

Should we make the API pod fail to start if migrations fail? If seed fails?

[bdunne]

I think it should fail to start the rails server if migrations fail.

[lindgrenj6]

I agree with @bdunne, I'm a big fan of having the pod fail x times and go into the CrashLoopBackoff state rather than coercing the user into thinking everything is mostly ok when the database hasn't been migrated.