Determine who will be the initial Watch Officers

[zachgoldfine44]

Description

We have a draft of what actions Watch Officers will take when an alert is identified, and what incident management system we will use to declare incidents. But who will the Watch Officers be? Platform team folks, and this is part of their existing on-call rotation? Product leads? Engineering leads? Portfolio leads? Other?

Definition of Done

• [x] List of initial Watch Officers identified (either names or roles)
• [x] Rotation defined (e.g., each person does it for one day? One week? New officers added or dropped? Any expectations on by when watch officer will have done, e.g., check by 1 pm?)

[zachgoldfine44]

Starting with @charwo , who may have some ideas or opinions to share, and @AparnaNittalaUSDS , who will think about the platform on-call process and how the Watch Officer practice might relate.

Check in on Wednesday 11/8.

[zachgoldfine44]

Update:

• What is on call? Only declaring / initiating an incident mechanism (e.g., PagerDuty, DataDog).
• If we want it to be more, we need to....(contract mod?)
• To what extent can Platform understand and perhaps investigate an issue vs needing an application team member with more domain knowledge?
• • In some cases, platform could understand or infer. In others, no.

Needs more thinking and discussion.

Steve: I don't think watch officers / on-call need to know much. Just see and record and declare an incident and reah out to the right people in the right place.

Chris:

• Needs to be centralized.
• It could be platform or a separate watch officer team, could be a mix of Feds and contractors, could come from any contract (like the support contract).
• If centralized, members cannot fix problems across all apps

Charles:

• Option 1: Add Slack process to platform on-call rotation
• • Seems most logical to Charles
• • Does not mean on-call person fixes the issue. Just that on-call person becomes aware of the issue.
• Option 2: Create new rotation (could be similar to platform on-call rotation, but not the same)
• Option 3: Engage enterprise command center and add Slack process to their on-call rotation.
• Option 4: robots

Brooke:

• Enterprise Command Center probably not a good option but they do have a mature process.

Mike Chelen:

• PagerDuty needs to be a key part of incident declaration.

Multiple people in chat:

• Watch officer rotation - all government Product Owners (or perhaps just all gov folks)

Next step:

• @AparnaNittalaUSDS to join @va-albers in doing a watch officer rotation tomorrow morning (Thurs 11/9)

[zachgoldfine44]

Update:

• Aparna did a walkthrough with Steve. Volunteering for a slot next week.
• Watch officer process not yet refined enough to be packaged and inherited by others
• Consider including current Platform on-call support team in walkthroughs to start learning and giving feedback on the process.
• Consider adding teammates from application teams to on-call support rotation.

Check back Wednesday 11/22

[zachgoldfine44]

Not just engineers. Definitely government employees.