nixos/mastodon: updates could be handled better

leo60228 commented 1 year ago

Issue description

When using the Mastodon NixOS module, updates currently work by the mastodon-init-db oneshot service running database migrations, and each individual Mastodon service being restarted. However, in some cases this can require manually restarting specific services, and in all cases it will lead to downtime.

The manual upgrade process recommended by upstream is very different, and should not result in downtime. It generally consists of restarting mastodon-sidekiq and mastodon-streaming (the latter only necessary for some releases), then reloading mastodon-web (which in the upstream unit sends SIGUSR1). Some releases may have additional steps. When this applies, it's usually running database migrations via SKIP_POST_DEPLOYMENT_MIGRATIONS=true bundle exec rails db:migrate before restarting/reloading services, and running bundle exec rails db:migrate afterwards.

While it wouldn't necessarily always work, and would still do some unnecessary steps for minor releases, I think it should be feasible for the module to do a zero-downtime update. The general template would likely be:

Have a unit run SKIP_POST_DEPLOYMENT_MIGRATIONS=true bundle exec rails db:migrate before restarting/reloading anything
Keep the current restart behavior for mastodon-sidekiq and mastodon-streaming
Set systemd.services.mastodon-web so that systemctl reload mastodon-web sends SIGUSR1 like the upstream unit
Set systemd.services.mastodon-web.reloadTriggers so that it gets reloaded instead of restarted
Have bundle exec rails db:migrate run once all three services have been reloaded/restarted successfully

I'm not fully sure how to implement this, but I think it should be possible.

Steps to reproduce

Update the Mastodon package when using the NixOS module.

Technical details

Please run nix-shell -p nix-info --run "nix-info -m" and paste the result.

 - system: `"aarch64-linux"`
 - host os: `Linux 5.15.76, NixOS, 22.05 (Quokka), 22.05.20220616.760ed64`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.9.1`
 - channels(root): `"nixos-22.05"`
 - nixpkgs: `/nix/store/1wiw1x4n29nzaj1sp7kqmwg1xnhskb3d-source`

leo60228 commented 1 year ago

Thinking about this more, one way to implement this might be to replace the current mastodon-init-db.service with an umbrella mastodon.service. Then, the other three services could be set to BindsTo and After mastodon.service so that they aren't newly started with an unmigrated database. For the proper behavior on updates, the three individual services could have restartIfChanged set to false, mastodon.service could have reloadTriggers set, and its reload script could orchestrate the update (run pre-deployment migrations, restart mastodon-sidekiq and mastodon-streaming, reload mastodon-web, and finally run post-deployment migrations).

leo60228 commented 1 year ago

Two issues are apparent with using SIGUSR1, but both are solvable:

SIGUSR1 is asynchronous, so it can't be known when the restart is completed. However, Puma (the web server implementation handling SIGUSR1) supports using a control socket instead: https://github.com/puma/puma#controlstatus-server
With Nix, the new Mastodon package will be at a different path. Puma's solution to this issue is that if the config file specifies the application directory (instead of having it default to the working directory) then that can be a symlink, and the symlink will be resolved at each restart: https://github.com/puma/puma/blob/master/docs/restart.md#additional-notes-1

NixOS / nixpkgs