EPIC Better system degradation monitoring

[lucyb]

GOAL: Improve our responsiveness to system problems so that we don’t rely on researchers reporting problems.

Tech support should be notified of important incidents with enough information to be able to investigate the problem.

Researchers should be provided with information about the current system status, so they don’t get frustrated and don’t need to contact tech support.

Measure

• System availability trends over time
• Number of incidents
• Tech support burden (survey?)

Solution

https://bennettoxford.freshstatus.io/

• Use this as a centralised place for updates
  • Send messages into slack when it’s updated
  • Users can sign up for email notifications for particular incidents
  • Automatically update when the database maintenance is taking place
  • Add future maintenance in when TPP notify us of scheduled maintenance windows
• Consider using names that represent the functions that researchers can perform, e.g releasing files, rather than application/tool names which might not mean as much
  • Be specific about the tasks that are affected - job submission, reviewing output, releasing files

[lucyb]

New issues: 1: ~Investigate feasibility of using freshstatus as a tool, including integrations. and integration with our services.~

1.a ~Get everyone setup on freshstatus, with seb as superadmin~

2. ~Start with https://github.com/opensafely-core/sysadmin/issues/36~

3. Then, integrating database maintenance mode (as "under maintance") Consider wording used, so that it's clear to users. Describe it as "TPP database availability"

4. ~#35~

5. ~VPN availability (stretch goal) - low value, as generally vpn has been up, but fails after the auth step, which we can't automate.~

6. ~Database availability check by job-runner (stretch goal)~

7. ~Move fresh ping alerts into #tech-support-channel https://github.com/opensafely-core/sysadmin/issues/61~

8. ~Integrate existing website fresh pings into the status page https://github.com/opensafely-core/sysadmin/issues/62~

9. ~Link to the status page in the relevant opensafely channels in slack~

10. ~Let researchers and the tech group know that the status page exists.~

11. ~Add fresh status to the tools and systems page in the Team Manual. Document how to set up an alert from freshping. https://github.com/opensafely-core/sysadmin/issues/63~

[lucyb]

Job runner should perform a generic db check Each component should be able to report its status

• Job runner (last seen), release hatch, db maintenance mode flags

This is picked up by job server and then scraped by freshping.

[benbc]

I've moved this epic into the pipeline here.