[TPP reboot] 2024-04-09

[madwort]

Checklist for TPP reboot, to be created when TPP inform us of a maintenance window:

1 hours downtime for the Level 2, 4 and Ubuntu server between 13:00 – 14:00 on Tuesday 9th

Preparation:

• [x] bring window start forward 10min, and extend window close by 30min, as TPP can be a bit optimistic about how long they need (=> window is 12:50-14:30)
• [x] schedule maintenance window in freshstatus
• [x] communicate advance notice of window to users on #opensafely and #opensafely-users channel
• [x] set reminder for window start

When window starts:

• [x] stop job runner
• [x] gracefully kill jobs

See the preparing for reboot section of the playbook for how to do this.

These commands can also be scheduled in advance if the maintenance window is out of hours, see the planned maintenance section for more details.

When window closes:

• [x] Check the VM is up and you can SSH in
  • If not chase TPP via email for an update.
  • It is likely they have not finished maintenance yet, if so you may need to extend window and communicate to users.

When the VM is back up:

• [x] Is docker up? run docker ps on the VM
• [x] Is job-runner up? check status, or sudo systemctl status jobrunner

If Level 4 was rebooted:

• [x] manually start release hatch on the level 4 VM
  • Note: once we've got the releases UI this will not be necessary

Finally

• [x] close window on freshstatus if still active.