website downtime during a deployment that includes certain types of db migration (seconds to several minutes)

[macolo]

problem statement

• website shows "db field does not exist" error during a deployment that includes a migration
• very likely this only applies to migrations that REMOVE one or several fields (see context below for further insights)

solutions / workarounds

• in general all such problems can be resolved by using a 2-step deployment approach.
• if you want to remove a db field, first remove any references to that field from the code, deploy. Then in a second commit add the migration that removes the field, deploy again.

Context

1. divio creates new container (including new code)
2. just after successful container creation, migrations are run on that container, at this point, new container is not live yet
3. database is now in migrated (new) state (amongst other cleanup) this might take a couple of minutes (i.e. S3 media headers check might take a while)
4. Meanwhile the website might make a database query based on the old code, running on the new (migrated) database state. This might lead to SQL errors.
5. after successsful "cleanup", divio control publishes the container
6. the website behaves now as expected

[viktor-yunenko]

Yes, it's common. Usually it happens during a short 2-6s window though, and this window might have been prolonged lately. That's why I was asking about a maintenance mode package a year ago.

[macolo]

Note: Issue description updated with more precise problem statement, context and solution approach.