search

mastodon / chart

Helm chart for Mastodon deployment in Kubernetes
GNU Affero General Public License v3.0
154 stars 89 forks source link

Run db:migrate in pre- and post-install/upgrade (#18, #26) #37

Open angdraug opened 1 year ago

angdraug commented 1 year ago

Not sure if it shows up for you, but GitHub tells me:

First-time contributors need a maintainer to approve running workflows. Learn more.

I don't think this PR is going to land until someone does the needful.

paolomainardi commented 1 year ago

Please can you release this PR? It seems ok. Otherwise, this chart cannot be used within a highly automated environment with Terraform + Helm; the helm installation never ends, and the migration job is never triggered by Helm, making it impossible to use it for a new fresh install.

cc @dunn

dunn commented 1 year ago

I'm not actually a maintainer of this repo, so I can't merge.

paolomainardi commented 1 year ago

I'm not actually a maintainer of this repo, so I can't merge.

Ops, so sorry, I saw your comments and just thought you was a maintainer too.

paolomainardi commented 1 year ago

Just tried this PR, and it doesn't work, the migrate job requires PVC already created otherwise the job cannot be executed.

The question is, does the migration job requires the PVC ?

paolomainardi commented 1 year ago

I tried again using a bucket instead of PVC, and now the problem is with the required redis instance, which should be up and running to finish the job.

paolomainardi commented 1 year ago

I tried running the migration job along with the other deployments, and it worked fine; it is the same approach the GitLab chart is taking. The concept is to let the scheduler restart the services until the migration job finishes to initialize the services; once finished, the pods start to come up and work fine.

Gitlab migration job used as a reference: https://docs.gitlab.com/charts/charts/gitlab/migrations/

renchap commented 1 year ago

Thanks for your work on this @paolomainardi!

I tried running the migration job along with the other deployments, and it worked fine; it is the same approach the GitLab chart is taking. The concept is to let the scheduler restart the services until the migration job finishes to initialize the services; once finished, the pods start to come up and work fine.

How would this work for version upgrades? The pre-upgrade migrations needs to be run before any of the new version application pods, otherwise those can generate errors on some requests (when trying to access a table that has not been migrated yet), while their /health endpoint returns OK (it does not check the schema version).

I worry it will create user-facing errors during the migration, or even a server to become unavailable if the migration does not happen and all pods are upgraded to the new version.

paolomainardi commented 1 year ago

@renchap yes, you're right; the issue with this approach is that users can face problems while migrations are running.

This issue can only be overcome by just using Helm, and the best choice is to avoid running them as is doing this chart and move most of the complexity to the application side.

Always looking how Gitlab does, they open sourced the database migration types they support: https://docs.gitlab.com/ee/development/migration_style_guide.html

The case for Mastodon is "Regular migrations" which according to their document must be always under 3 minutes if higher than must be moved on post-deployment or background migrations.

Is not very clear indeed, what it happens during the 3-minutes window, maybe the migrations are always written in a way that prev/next releases are always compatible.

This is migration helm chart documentation: https://docs.gitlab.com/charts/charts/gitlab/migrations and from my direct experience, they run along the other deployments.

jessebot commented 1 year ago

is there any chance this could be moved forward?