Configure Pager Duty

[paulsackley-nava]

VRO engineers have to currently volunteer to have an informal process for being monitors for SecRel and Dependabot. Also individual engineers are siloed in some instances for a POC between partner teams. Formalizing this process will allow people to fairly distribute the work and prevent a single point of failure and knowledge on the team. This will also allow our partner teams to feel more supported when reaching out because instead of a person who may not be available there is a role that can help or be a liaison for help.

Acceptance Criteria

1. There is an oncall rotation process that distributes a primary/secondary fairly and weekly configured in PagerDuty by sprint
2. All engineers have an agreed place to put their availability that pager duty can use to update the schedule
3. The team is able to see who is on call via Pager Duty
4. There is a method for engineers to swap oncall weeks in cases of conflicts or emergencies.
5. On call should be within normal working business hours @BerniXiongA6 to confirm

Notes about work https://app.zenhub.com/workspaces/vro-team-6557e67173391c000e1409f3/issues/gh/department-of-veterans-affairs/abd-vro/2570 https://app.zenhub.com/workspaces/vro-team-6557e67173391c000e1409f3/issues/gh/department-of-veterans-affairs/abd-vro/2889

[meganhicks]

break out this into next sprint-There is an agreed to comms, partner team SLA and demo of the process ready for the next partner team meeting and move 5 AC to incident response: 5. Documentation is added for how the oncall rotation works and what the on call person supports. For specifics on incident response link to the work done last sprint, see ticket below.

[meganhicks]

add alerting recommendations of who is on call

[tejans24]

In-order to implement "The team is alerted who is on call in the benefits-vro slack channel", "Pager duty can make calls to the team calendar to grab events data for engineer time off":

• We'll need to add a github action to use PagerDuty API
• This requires a oauth scoped key to use the API

@BerniXiongA6 it says "Only account owners ors that administer or get data from PagerDuty via REST API. Learn more admins can manage Scoped OAuth clients.". Can we contact the admin to create a client for VRO?

[BerniXiongA6]

Thanks @tejans24 -- I can tag the person/s that helped us get access to PagerDuty to see what they know/say about this! cc: @meganhicks

[lisac]

@tejans24 : for awareness, i'm working on getting PagerDuty access. Submitted a request (DOTSD-49225) on May 15 and nudged the ticket today.

[BerniXiongA6]

Here's the Slack thread containing my request for assistance! https://dsva.slack.com/archives/CBU0KDSB1/p1716309272574099?thread_ts=1708619451.187059&cid=CBU0KDSB1

[lisac]

update: I got access to PagerDuty today and am now on the Benefits VRO team. (hat tip to Berni for the extra help getting my access request fielded - merci!)

[meganhicks]

@tejans24 do you think we need to demo this new tool to the team?

[lisac]

demo-ed on 6/24: https://dsva.slack.com/archives/C04PKJ7FQCE/p1719247630420849