Report Major VA.gov Incidents Following VA OIT's Major Incident Management (MIM) Process

[raywangoctova]

LOE

Large -- there are a lot of unknowns and SNOW seems like a very complex system that is reliant on information being input in very specific ways (based on our call and demo with a power user of SNOW). We will have to start out with discovery that gives us more information about how to integrate. We also need some information from OCTODE about when escalations are necessary

Problem Statement

• VA requires all teams within the OIT organization to follow the Major Incident Management (MIM) process for notification, escalation, communication, and response for all Major Incidents.
• A major incident is a high-impact, high-urgency incident that affects many users, depriving the business of one or more crucial services, such as patient care, benefits processing, and cemetery operations.
• An incident becomes major when it significantly disrupts the business and demands a response beyond the routine incident management process.
• Due to this VA OIT requirement, the VA.gov Platform team needs to update the VA.Gov Incident Playbook to update guidance and enables the technical support team to perform the approved MIM report methods.

User Impact

This project will impact the platform team's overall response to any major incidents on the VA.gov platform.

Where was this problem reported?

• Rod Kearns, Director, Incident and Problem Management (IPM), Rodney.Kearns@va.gov, (512) 638-1287
• OCTO: CTO and Deputy CTO for Digital Experience

How well do we understand the problem?

From Rod Kearns

• How should VA.gov ops team report a major incident (Phone call and press 9, or filing a Tier 1 or 2 SNOW ticket, most likely?)
• Option 9 is always the fastest route to establishing a MI. Since VA.gov is a different kind of customer, we would likely develop a desk reference for our MIM team to facilitate rapid validation and acceptance.
• How can we add the right visibility of an HPI/CPI without slowing down the work to fix the problem?
• In cases like this where our on-call team is reporting an issue, the overhead of creating a MIM bridge and getting ECC up to speed would not have helped resolve the problem faster since the right team (the VA.gov on-call team) had the issue and was working it. However, in other cases (i.e. if the problem is with a downstream system like MPI), perhaps establishing a CPI/HPI might be a good way to get eyes from other teams on the problem.
• I would suggest that we borrow from how we operate within the EHRM Joint architecture, in that VA.gov would alert us to the incident, we validate impact as warranting a MI (we will likely need to establish specific criteria I'd think), and we send "Comms" in the form of our usual notification. We would either use an existing VA.gov bridge or create one for coordination/update purposes. We can state that "no additional resources are required at this time" if there is no need for others to join the call. We might consider establishing a specific Leadership Awareness DL, so as not to inadvertently draw in resources better used elsewhere. We could always escalate to a full Major Incident call and usual HPI/CPI DL if needed while maintaining the ongoing bridge.

Additional discovery is needed with the IPM and Platform teams about process integration, dependencies, and handoffs between the MIM process and VA.gov Incident Response Playbook.

Current documentation:

• VA.gov Incident Response Playbook
• https://vfs.atlassian.net/wiki/spaces/ECP/pages/1565294641/VA.Gov+Incident+Playbook
• Major Incident Management Process Flowchart
• https://dvagov.sharepoint.com/sites/OITProcessAssetLibrary/_layouts/15/viewer.aspx?sourcedoc={6F78BC98-7ED5-42AE-A43A-6293DE3B4349} -https://app.zenhub.com/files/133843125/0ce04160-9d39-4962-9357-4258ff27d311/download
• VA Major Incident Management Process
• https://dvagov.sharepoint.com/sites/OITProcessAssetLibrary/_layouts/15/viewer.aspx?sourcedoc={3C99AB3A-0142-4F73-AF45-919CA9ABC675} -https://app.zenhub.com/files/133843125/21c4a4b4-0e00-4f7d-91a8-6e404c2af8a6/download

What is the acceptance criteria?

• VA.gov Incident Respond Playbook complies with the MIM Process.
• Present the applicable updates on VA.gov Incident Respond Playbook to the IPM leadership for obtaining confirmation of compliance.
• Conduct at least one tabletop exercise of a major incident that follows the new process.

How should we measure success?

• 100% of major VA.gov incidents follow the updated VA.gov Incident Respond Playbook.

TODOs

• [x] - Convert this issue to an epic
• [ ] - Assign project team
• [ ] - Identify this project's OCTO and Vendor key POCs
• [ ] - Identify additional teams to be part of this project's acceptance
• [ ] - Identify project target end date
• [ ] - Conduct project kickoff meeting
• [ ] - Develop and implement a project communication plan

[raywangoctova]

Per discussion at the onsite workshop for 2023 platform objectives, Cory indicated that he would follow up on this project request for the next steps.

[little-oddball]

Per request:

There is some decent information around the request and with the general notion of Platform being compliant, that should add some clear / concrete boundaries to work within. As this ticket is currently written, there is a lot of unknown(s) from a Platform team perspective so a fair amount of discovery will be required to flush out the specifics, to get educated and get detailed on what items Platform needs and actions they should take to get there. This feels like a pretty good size effort and needs very clear MVPs identified and defined. I believe the AC is a bit high level and needs more added to it. For example, I would assume Platform documentation would need to be updated and communicated to Platform and VFS teams. I would also say conducting a tabletop exercise shouldn’t be an AC but making part of the process and possibly even scheduling the tabletop could (shouldn’t be holding the item success based on a future event that there is no control over).

So to me, step #1 is to put a sprint of Discovery into play and work to iron out and harden the specifics, deliverables, etc. The output should be a clear path forward with very clear and specific AC with small bitesize and testable deliveries.

[jwoodman5]

I don't really understand this whole process, but do think I understand what the A/C should be. I feel like there would be fair bit of discovery that would happen along the way as the playbook is updated.

[AparnaNittalaUSDS]

This process is being formulated as part of the code Yellow, in collaboration with the ECC. Datadog monitor and ServiceNow integration is in place so that datadog alerts with Priority P1 (SEV-1 classification in SNOW) are available in ECC. Still there are few questions and processing pending discussion and process refinement on how many of these P1 alerts are to be handled through CPI/HPI MIM process of OIT.

Here is the doc that @acrollet has put together - https://dvagov-my.sharepoint.com/:w:/g/personal/adrian_rollett_va_gov/EY-JhFIzFOFOkBSvo5QE81ABtgVMxunyeS5f03ebynT49w?e=DX1Con

[annekerr49]

Current status: We have an initial POC to integrate Datadog monitors into SNOW for usage by ECC Event Management. Next followup steps needed are: Update alert message with required response action (contact Platform oncall? declare MIM?) Configure alerting integration for all mission critical monitors (all P1/P2?) Run tabletop exercise with ECO/EM/OTG to confirm successful incident response (make sure everyone has necessary access & understands procedures)

[AparnaNittalaUSDS]

@AparnaNittalaUSDS to check what has been done and what is remaining with the OIT integration.. outline items like documentation, tagging standards etc

[chrisj-usds]

Can anyone comment on where we are with this effort? @JeffKeeneVAGov @BillChapmanUSDS