[database-migrations] Run it as a precursor to application deployment

[Pothulapati]

Is your feature request related to a problem? Please describe

Currently, WebApp changes that include database migrations are done in two separate PR's (i.e database migration change and the webapp changes) while making sure that the database migration is deployed first and then the webapp changes.

Describe the behaviour you'd like

The above process includes mental overhead which is unnecessary for a lot of database changes. This can be fixed by making sure the database migrations are first deployed before the application rollout happens. This can be done in the following aways:

• Use ArgoCD's Resource hooks to make sure db-migration job is done, and the application rollout happens only after a success.
• Update the installer to have the db-migration job as ainit-container in the server.

Additional context

• https://www.notion.so/gitpod/WebApp-Rollout-Upgrades-Sync-586c4fc6086741119484342cf5abd59e

[Pothulapati]

Update:

We already seem to do that in werft deploy pipeline and not as another application as part of Gitpod Application in Argo. With this, I don’t see a reason for using the pre-hook thing (as werft pipeline gives the same properties i.e not deploying the application if the database migration failed) unless we want to get rid of something like this from werft.

While moving into the installer might make sense, It has more problems around rollouts as the application has already rolled out by the time db-migrations is failing, making rollbacks complex.

[Pothulapati]

@geropl With the above update, We don't have a concrete path forward as db-migrations already runs as a pre-cursor to application deployment through werft. Moving db-migrations either to argocd or to installer does not give us any benefits in the short term as the behavior would be the same.

If you think there is something more or different we could do here, Feel free to communicate the same. I will have to close the issue otherwise.

Thanks :)

[corneliusludmann]

Discussed this with Gero this week. The primary issue is the need to better coordinate rollouts of API changes (e.g. DB schema changes). Maybe, we need to tag commits that need to be deployed first before we can continue updating the application. Before we don't know exactly what the new deployment pipeline looks like, it does not make sense to work on this.

I'm closing this issue for now. When we know what exactly we need, we can open a new issue for this.