CMS Incident Response Protocol

[daddison48]

Story/Issue

As a user of the CMS I need to know when their are potential system issues or outages and the workarounds that affect my use of the system so that I can use workarounds to continue my work.

Background

On Friday (12/11/2020) CMS team became aware of a PIV login issue, but users were not notified until Tuesday (12/15) of the issue and the workaround to login without PIV. Dave C. expressed concern over the fact that users were not notified outside of slack, as most new users onboarding will not be slack users.

Working group for this should be @oksana-c @olivereri @erogray @daddison48 (optional-ooo) @cmaeng @VanessaLuxen @indytechcook

ACs

• [x] Establish our Incident Response plan for the team to follow. There is an initial framework in the helpdesk workflow docs: https://docs.google.com/document/d/1piIkYdFjvcy0_OMlD3oWVj9o6sX3g5ykG0bopGMiHxI/edit#heading=h.da5y89axy2f0
  • [x] Should include creating a JSD ticket
  • [ ] Should include communication requirements (CMS alert, email, slack, etc)
  • [x] Should include instructions for Incident Commanders
• [ ] Publish plan publicly
• [ ] Ensure CMS team is aware of the plan, how to follow it, and their role.
• [x] Need to ensure the CMS Admin distro list (cms-admin@va.gov) is working for email notifications to the CMS team - @cmaeng

[erogray]

Some useful CivicActions documents exist, which can probably be usefully adapted: https://handbook.civicactions.com/en/latest/100-security/incidents/ https://handbook.civicactions.com/en/latest/100-security/incident-response-plan/ https://handbook.civicactions.com/en/latest/100-security/incident-response-checklist/

[erogray]

@erogray to look for contractual language around critical defects (in base contract) — and language around "Critical incidents" in QASP document

[erogray]

can publish to a new subdirectory of https://github.com/department-of-veterans-affairs/va.gov-team/tree/master/platform/cms

[erogray]

https://docs.google.com/document/d/1QhRZ-mBF2-EEtjgmcUxAsdk_de3jL3jLESATOACGjRs/edit# seems like a good start — I have a couple questions about communications to stakeholders that are probably best answered by someone else on the team (perhaps @daddison48 or @cmaeng ) that are highlighted in the document. This is ready for review by someone besides myself and Vanessa.

[erogray]

@daddison48 would it be helpful to just schedule a meeting to talk through this?

[daddison48]

I provided some thoughts and input in the doc. I think we may need to work towards a v1 of this plan and then iterate on a v1.1 as we make decisions around security incidents, oncall, ISSO, etc.