[Platform SRE] Update the incident response process

[jhouse-solvd]

Problem Statement

The incident response process is unclear. It is unclear when to treat an issue as an incident. And, it is unclear how to classify, prioritize, and respond to incidents when they occur.

Background / Context

• There is likely some existing documentation, but it can be improved, condensed, and/or better connected
• The goal is to make the platform's incident response process into a service, ie
  • There should be a process for users needing to declare an incident
  • There should be a playbook for incident support personnel to follow

How might we

...update the incident response process so that it's clear for end-users? ...update the incident response process so that it's clear for SRE personnel? ...update the incident response process with guidance for classification and prioritization? ...update the incident process with communication protocol and responsibilities?

Hypothesis or Bet

This will make it easier for VFS teams to declare an incident. This will make it easier for SRE (and platform teams) to respond to incidents.

We will know we're done when... ("Definition of Done")

When there is an updated incident response process in place.

Known Blockers/Dependencies

List any blockers or dependencies for this work to be completed

Projected Launch Date

TBD

Launch Checklist

Is this service / tool / feature...

... tested?

• [ ] Usability test (TODO: link) has been performed, to validate that new changes enable users to do what was intended and that these changes don't worsen quality elsewhere. If usability test isn't relevant for this change, document the reason for skipping it.
  • [ ] ... and issues discovered in usability testing have been addressed.
  • Note on skipping: metrics that show the impact of before/after can be a substitute for usability testing.
• [ ] End-to-end manual QA or UAT is complete, to validate there are no high-severity issues before launching
• [ ] (if applicable) New functionality has thorough, automated tests running in CI/CD

... documented?

• [ ] New documentation is written pursuant to our documentation style guide
• [ ] Product is included in the List of VSP Products
  • List the existing product that this initiative fits within, or add a new product to this list.
• [ ] Internal-facing: there's a Product Outline
• [ ] External-facing: a User Guide on Platform Website exists for this product/feature tool
• [ ] (if applicable) Post to #vsp-service-design for external communication about this change (e.g. VSP Newsletter, customer-facing meetings)
• [ ] Update Emergency Operating Procedures: https://github.com/department-of-veterans-affairs/va.gov-team-sensitive/blob/master/OnCall/Emergency%20Operating%20Procedures.md

... measurable

• [ ] (if applicable) This change has clearly-defined success metrics, with instrumentation of those analytics where possible, or a reason documented for skipping it.
  • For help, see: Analytics team
• [ ] This change has an accompanying VSP Initiative Release Plan.

When you're ready to launch...

• [ ] Conduct a [go/no-go] (https://vfs.atlassian.net/wiki/spaces/AP/pages/1670938648/Platform+Crew+Office+Hours#Go%2FNo-Go) when you're almost ready to launch.

Required Artifacts

Documentation

• PRODUCT_NAME: directory name used for your product documentation
• Product Outline: link to Product Outline
• User Guide: link to User Guide

Testing

• Usability test: link to GitHub issue, or provide reason for skipping
• Manual QA: link to GitHub issue or documented results
• Automated tests: link to tests, or "N/A"

Measurement

• Success metrics: link to where success metrics are measured, or where they're defined (Product Outline is OK), or provide reason for skipping
• Release plan: link to Release Plan ticket

[jhouse-solvd]

Existing oncall rotation doc that contains links to incident categorization, incident command info, incident response playbook

https://github.com/department-of-veterans-affairs/va.gov-team-sensitive/tree/master/OnCall

[jhouse-solvd]

One of the first things that we need to do is to understand what exists already.

We can refer to the documentation posted above and try to wrap our understanding around existing processes and information.

[jhouse-solvd]

It could be worth it to workshop incident categorization and prioritization based on actual incidents that have occurred over the past couple of years.

[zachclarity]

@jhouse-solvd Which of the following should be covered by the Incident Plan ? https://docs.google.com/spreadsheets/d/1Fn2lD419WE3sTZJtN2Ensrjqaz0jH3WvLaBtn812Wjo

[oseasmoran73]

A couple of questions:

• Is the proposed workflow the following:
  • Pagerduty triggered --> Infra notified --> (in the event we need to coordinate with other teams) escalate to SRE --> SRE triages/escalates accordingly
• I believe for platform on-call non-critical, if they were to escalate, it would escalate to Infra. Would non-critical default to infra, or would it go to SRE?