TPP reboot: 14th Sept 11pm

[lucyb]

TPP (Chris) have informed us that they are planning a reboot of OpenSafely servers tonight at 11pm. Chris will check everything is back up as expected when he's back in the office tomorrow morning.

Preparation:

• [ ] bring window start forward 10min, and extend window close by 30min, as TPP can be a bit optimistic about how long they need.
• [x] schedule maintenance window in freshstatus (TODO: how?)
• [x] communicate advance notice of window to users on #opensafely and #opensafely-users channel (possibly freshstatus may be able to do this for us?)
• [ ] set reminder for window start (again freshping might be able to do this).

When window starts:

• [ ] stop job runner: https://github.com/opensafely-core/backend-server/blob/main/jobrunner/playbook.md#startingstopping-the-service
• [ ] gracefully kill jobs: https://github.com/opensafely-core/backend-server/blob/main/jobrunner/playbook.md#preparing-for-reboot

When window closes:

• [x] Check the VM is up and you can SSH in
  • If not chase TPP via email for an update.
  • It is likely they have not finished maintenance yet, if so you may need to extend window and communicate to users.

When the VM is back up:

• [x] Is docker up? run docker ps on the VM.
• [x] Is job-runner up? check status, or sudo systemctl status jobrunner.
• [x] manually start release hatch on the level 4 VM
  • Note: once we've got the releases UI this will no be nessecary
• [x] manualy start level3/VM rsync (TODO: write this up in playbook)

Finally

• [x] close window on freshstatus if still active.