service/synapse: Initial setup

[e1mo]

Secrets for production not yet added to avoid breaking our setup in case we were to accidentally deploy this.

Migration plan (proposal)

• [x] Ensure new server (hamilton?) is up and running and being monitored, deploy without synapse (2-n days prior)
• [x] Announce downtime in Allgemein, Große Halle and administration (1 day prior)
• [x] Reduce DNS TTL to 900s (6h to 1 day prior)
• [x] Announce imminent downtime in Allgemein and administration (A few minutes prior to deployment)
• [x] Reduce DNS TTL to 300s
• [x] Stop and mask marix-synapse.service service on old host (hopper, sudo systemctl disable --now matrix-synapse.service; sudo systemctl mask matrix-synapse.service)
  • Could only disable and not mask the unit (file path foo)
• [x] Shut old server down and take a manual snapshot (just in case)
• [x] Deploy new server with services.matrix-synapse = lib.mkForce false; (but with postgres enabled)
• [x] Switch DNS over to new server
• [x] Detach matrix-media-repo volume from old server and boot it back up
• [x] Sync database directly to new host: sudo -u postgres pg_dump --format=custom synapse | ssh hamilton.mon.net.chaos.jetzt "sudo -u postgres pg_restore -d synapse --format=custom --single-transaction --verbose" (needs SSH with agent forwarding) Database is to big for dumping and importing separately
  • sudo -u postgres pg_dump --format=custom synapse | ssh e1mo@hamilton.mon.net.chaos.jetzt "sudo -u postgres pg_restore -d matrix-synapse --format=custom --single-transaction --verbose --no-owner --role=matrix-synapse" (Wrong db name on target, and didn't account for different db usernames on target)
• [x] Ensure matrix secrets on new host match old host
• [x] (Only afer db migration is finished) Re-deploy new host with services.matrix-synapse = true; and log level of info
• [x] Adjust owner of synapse_media_store files (different uids on the servers): chown matrix-synapse:matrix-synapse /mnt/synapse_media_store -R
• [x] Monitor logs of matrix synapse for problems
• [x] Validate basic functionality (sending messages, upload files, check federation is working)
• [x] Ensure all configuration changes are commited to the repo, prolong DNS TTL
• [x] Send end of downtime message to all channels

_Based on the last migration_

[e1mo]

The build is failing since the secrets (e.g. synapse signing key) for prod are not yet present (to avoid messing up federation and stuff if we were to accidentally deploy).

[e1mo]

As soon as #26 is merged, I'll re-write this to be for hamilton. However, the migration plan is already at an usable state, so I'd be happy for some feedback regarding it.