Open ph-One opened 2 years ago
.cc @td-usds
@little-oddball @andreahewitt-odd @mchelen I would like to prioritize this requirement to be completed in Q2 FY23. Should this be treated as a project or something small enough to addressed as a tech debt by the support team?
From: Kearns, Rodney T. [Rodney.Kearns@va.gov](mailto:Rodney.Kearns@va.gov) Sent: Wednesday, July 27, 2022 4:36 PM To: Worthington, Charles (he/him/his) [Charles.Worthington@va.gov](mailto:Charles.Worthington@va.gov) Cc: Cunningham, Robert M. (OI&T) [Robert.Cunningham2@va.gov](mailto:Robert.Cunningham2@va.gov); Carbajal, Rogaciano [Rogaciano.Carbajal@va.gov](mailto:Rogaciano.Carbajal@va.gov); Chelen, Michael N. [Michael.Chelen@va.gov](mailto:Michael.Chelen@va.gov); Johnston, Christopher M. (he/him/his) [Christopher.Johnston2@va.gov](mailto:Christopher.Johnston2@va.gov); Trimm, Cory D. (he/him/his) [Cory.Trimm@va.gov](mailto:Cory.Trimm@va.gov); Vogt, Randall C. [Randall.Vogt@va.gov](mailto:Randall.Vogt@va.gov) Subject: RE: Connecting VA.gov Platform Ops to ECC
Charles,
Happy to support a call to discuss at any time of your convenience. In the meantime, I’ll offer the following…
Questions to answer: • How should VA.gov ops team report a major incident (Phone call and press 9, or filing a Tier 1 or 2 SNOW ticket, most likely?) o Option 9 is always the fastest route to establishing a MI. Since VA.gov is a different kind of customer, we would likely develop a desk reference for our MIM team to facilitate rapid validation and acceptance. • How can we add the right visibility of an HPI/CPI without slowing down the work to fix the problem? In cases like this where our on-call team is reporting an issue, the overhead of creating a MIM bridge and getting ECC up to speed would not have helped resolve the problem faster since the right team (the VA.gov on-call team) had the issue and was working it. However, in other cases (i.e. if the problem is with a downstream system like MPI), perhaps establishing a CPI/HPI might be a good way to get eyes from other teams on the problem. o I would suggest that we borrow from how we operate within the EHRM Joint architecture, in that VA.gov would alert us to the incident, we validate impact as warranting a MI (we will likely need to establish specific criteria I’d think), and we send “Comms” in the form of our usual notification. We would either use an existing VA.gov bridge or create one for coordination/update purposes. We can state that “no additional resources are required at this time” if there is no need for others to join the call. We might consider establishing a specific Leadership Awareness DL, so as not to inadvertently draw in resources better used elsewhere. We could always escalate to a full Major Incident call and usual HPI/CPI DL if needed while maintaining the ongoing bridge.
This is all a bit off the cuff and likely need to work through some details specific to VA.gov, but I working a process that addresses the need is certainly doable.
ATB, Rod
Rod Kearns Director, Incident and Problem Management (IPM) Office of Information and Technology, Technology Development, Security & Operations Enterprise Command Operations (ECO) GFE Mobile: (512) 638-1287
From: Worthington, Charles (he/him/his) [Charles.Worthington@va.gov](mailto:Charles.Worthington@va.gov) Sent: Wednesday, July 27, 2022 10:46 AM To: Chelen, Michael N. [Michael.Chelen@va.gov](mailto:Michael.Chelen@va.gov); Johnston, Christopher M. (he/him/his) [Christopher.Johnston2@va.gov](mailto:Christopher.Johnston2@va.gov); Trimm, Cory D. (he/him/his) [Cory.Trimm@va.gov](mailto:Cory.Trimm@va.gov); Kearns, Rodney T. [Rodney.Kearns@va.gov](mailto:Rodney.Kearns@va.gov); Vogt, Randall C. [Randall.Vogt@va.gov](mailto:Randall.Vogt@va.gov) Cc: Cunningham, Robert M. (OI&T) [Robert.Cunningham2@va.gov](mailto:Robert.Cunningham2@va.gov) Subject: Connecting VA.gov Platform Ops to ECC
Hi Team – Bob and I were talking about our desire to explore how best to connect our VA.gov operations team with the ECC team.
For context, VA.gov (and the underlying platform tech ecosystem) has a 24/7 “on-call” pager rotation which responds to issues with VA.gov when they arise. On rare occasions, there is an issue that probably rises to the level of “HPI” or “CPI.” We had one of these recently, where a deploy failed and caused a cascading problem. The on-call VA.gov team ID’ed the problem using monitoring and fixed it in about 90 minutes. Most likely, this should have led to an HPI or CPI, at least so we could track the incident and others in VA and the help desk would be able to know it was ongoing and being worked.
Questions to answer: • How should VA.gov ops team report a major incident (Phone call and press 9, or filing a Tier 1 or 2 SNOW ticket, most likely?) • How can we add the right visibility of an HPI/CPI without slowing down the work to fix the problem? In cases like this where our on-call team is reporting an issue, the overhead of creating a MIM bridge and getting ECC up to speed would not have helped resolve the problem faster since the right team (the VA.gov on-call team) had the issue and was working it. However, in other cases (i.e. if the problem is with a downstream system like MPI), perhaps establishing a CPI/HPI might be a good way to get eyes from other teams on the problem.
Perhaps a good next step would be phone call amongst those on the “to:” line?
Thank you! Charles
@little-oddball @andreahewitt-odd @mchelen I would like to prioritize this requirement to be completed in Q2 FY23. Should this be treated as a project or something small enough to addressed as a tech debt by the support team?
@raywangoctova Sorry for the delay, I would treat this as a project.
Prioritizing below https://github.com/department-of-veterans-affairs/va.gov-team/issues/48270
Description
As platform infrastructure, we need to create a ServiceNow (SNOW) ticket for each GitHub release.
Background/context
Technical notes
Tasks
Acceptance Criteria