search

department-of-veterans-affairs / va.gov-team

Public resources for building on and in support of VA.gov. Visit complete Knowledge Hub:
https://depo-platform-documentation.scrollhelp.site/index.html
281 stars 197 forks source link

[ServiceNow] Automate Change Requests for Releases in ServiceNow #42029

Open ph-One opened 2 years ago

ph-One commented 2 years ago

Description

As platform infrastructure, we need to create a ServiceNow (SNOW) ticket for each GitHub release.

Background/context

Technical notes

Tasks

Acceptance Criteria

ph-One commented 2 years ago

.cc @td-usds

raywangoctova commented 1 year ago

@little-oddball @andreahewitt-odd @mchelen I would like to prioritize this requirement to be completed in Q2 FY23. Should this be treated as a project or something small enough to addressed as a tech debt by the support team?

raywangoctova commented 1 year ago

From: Kearns, Rodney T. [Rodney.Kearns@va.gov](mailto:Rodney.Kearns@va.gov) Sent: Wednesday, July 27, 2022 4:36 PM To: Worthington, Charles (he/him/his) [Charles.Worthington@va.gov](mailto:Charles.Worthington@va.gov) Cc: Cunningham, Robert M. (OI&T) [Robert.Cunningham2@va.gov](mailto:Robert.Cunningham2@va.gov); Carbajal, Rogaciano [Rogaciano.Carbajal@va.gov](mailto:Rogaciano.Carbajal@va.gov); Chelen, Michael N. [Michael.Chelen@va.gov](mailto:Michael.Chelen@va.gov); Johnston, Christopher M. (he/him/his) [Christopher.Johnston2@va.gov](mailto:Christopher.Johnston2@va.gov); Trimm, Cory D. (he/him/his) [Cory.Trimm@va.gov](mailto:Cory.Trimm@va.gov); Vogt, Randall C. [Randall.Vogt@va.gov](mailto:Randall.Vogt@va.gov) Subject: RE: Connecting VA.gov Platform Ops to ECC

Charles,

Happy to support a call to discuss at any time of your convenience. In the meantime, I’ll offer the following…

Questions to answer: • How should VA.gov ops team report a major incident (Phone call and press 9, or filing a Tier 1 or 2 SNOW ticket, most likely?) o Option 9 is always the fastest route to establishing a MI. Since VA.gov is a different kind of customer, we would likely develop a desk reference for our MIM team to facilitate rapid validation and acceptance. • How can we add the right visibility of an HPI/CPI without slowing down the work to fix the problem? In cases like this where our on-call team is reporting an issue, the overhead of creating a MIM bridge and getting ECC up to speed would not have helped resolve the problem faster since the right team (the VA.gov on-call team) had the issue and was working it. However, in other cases (i.e. if the problem is with a downstream system like MPI), perhaps establishing a CPI/HPI might be a good way to get eyes from other teams on the problem. o I would suggest that we borrow from how we operate within the EHRM Joint architecture, in that VA.gov would alert us to the incident, we validate impact as warranting a MI (we will likely need to establish specific criteria I’d think), and we send “Comms” in the form of our usual notification. We would either use an existing VA.gov bridge or create one for coordination/update purposes. We can state that “no additional resources are required at this time” if there is no need for others to join the call. We might consider establishing a specific Leadership Awareness DL, so as not to inadvertently draw in resources better used elsewhere. We could always escalate to a full Major Incident call and usual HPI/CPI DL if needed while maintaining the ongoing bridge.

This is all a bit off the cuff and likely need to work through some details specific to VA.gov, but I working a process that addresses the need is certainly doable.

ATB, Rod

Rod Kearns Director, Incident and Problem Management (IPM) Office of Information and Technology, Technology Development, Security & Operations Enterprise Command Operations (ECO) GFE Mobile: (512) 638-1287

From: Worthington, Charles (he/him/his) [Charles.Worthington@va.gov](mailto:Charles.Worthington@va.gov) Sent: Wednesday, July 27, 2022 10:46 AM To: Chelen, Michael N. [Michael.Chelen@va.gov](mailto:Michael.Chelen@va.gov); Johnston, Christopher M. (he/him/his) [Christopher.Johnston2@va.gov](mailto:Christopher.Johnston2@va.gov); Trimm, Cory D. (he/him/his) [Cory.Trimm@va.gov](mailto:Cory.Trimm@va.gov); Kearns, Rodney T. [Rodney.Kearns@va.gov](mailto:Rodney.Kearns@va.gov); Vogt, Randall C. [Randall.Vogt@va.gov](mailto:Randall.Vogt@va.gov) Cc: Cunningham, Robert M. (OI&T) [Robert.Cunningham2@va.gov](mailto:Robert.Cunningham2@va.gov) Subject: Connecting VA.gov Platform Ops to ECC

Hi Team – Bob and I were talking about our desire to explore how best to connect our VA.gov operations team with the ECC team.

For context, VA.gov (and the underlying platform tech ecosystem) has a 24/7 “on-call” pager rotation which responds to issues with VA.gov when they arise. On rare occasions, there is an issue that probably rises to the level of “HPI” or “CPI.” We had one of these recently, where a deploy failed and caused a cascading problem. The on-call VA.gov team ID’ed the problem using monitoring and fixed it in about 90 minutes. Most likely, this should have led to an HPI or CPI, at least so we could track the incident and others in VA and the help desk would be able to know it was ongoing and being worked.

Questions to answer: • How should VA.gov ops team report a major incident (Phone call and press 9, or filing a Tier 1 or 2 SNOW ticket, most likely?) • How can we add the right visibility of an HPI/CPI without slowing down the work to fix the problem? In cases like this where our on-call team is reporting an issue, the overhead of creating a MIM bridge and getting ECC up to speed would not have helped resolve the problem faster since the right team (the VA.gov on-call team) had the issue and was working it. However, in other cases (i.e. if the problem is with a downstream system like MPI), perhaps establishing a CPI/HPI might be a good way to get eyes from other teams on the problem.

Perhaps a good next step would be phone call amongst those on the “to:” line?

Thank you! Charles

little-oddball commented 1 year ago

@little-oddball @andreahewitt-odd @mchelen I would like to prioritize this requirement to be completed in Q2 FY23. Should this be treated as a project or something small enough to addressed as a tech debt by the support team?

@raywangoctova Sorry for the delay, I would treat this as a project.

mchelen-gov commented 1 year ago

Prioritizing below https://github.com/department-of-veterans-affairs/va.gov-team/issues/48270