Top Use Cases: Incident Reports - Research, Ideation, Testing

[bianca-rivera]

Description

When partner teams escalate an issue or current deployment blocker, they rely on posting on the #benefits-vro-support channel. However, the VRO team does not have designated steps for how to communicate with partner teams regarding 1) next steps for addressing the issue, 2) when the VRO team will provide status updates for resolving the issue and 3) documentation of post-mortem and lessons learned. There is a need to create a simple, automated workflow that partner teams can use and that VRO engineers can rely on.

Why

So we can set feasible expectations with partner teams and ensure we are providing consistent support processes moving forward

AC

• [ ] document current state of how partner teams are have been escalating incidents
• [ ] assessment of if/how partner team engineers are using other non-designated pathways (e.g. DM a VRO engineer)
• [x] research and develop ideas workflow for reporting incidents
• [x] test process internally and if possible, with a partner team engineer
• [x] develop a refined process and SLA - update - completed by Incident Response

Notes

Resources

reference work done in #2570 Incident Response

[lisac]

update on pagerduty -> slack integration: rather than provision a "bot" pagerduty account, am exploring this path: https://support.pagerduty.com/docs/slack-user-guide#create-an-incident-trigger-slack-workflow . relevant:

This option is useful for employees who do not have PagerDuty licenses, but still need to be able to trigger PagerDuty incidents in Slack.

update (14:54 ET): realized that ^ describes what was implemented earlier this week. as part of testing, we'll want to ask someone who does not have a pagerduty account to initiate the slack workflow. as part of that test, we should alert the on-call engineer and also consider scheduling a maintenance window in pagerduty - from what i understand, this will prevent notifications from being pushed to the on-call engineer.

[bianca-rivera]

demoed workflow to VRO team on Tuesday, July 9 to get feedback; will demo to partner teams on Thursday, July 11 during Partner Team Time