cds-snc / notification-planning

Project planning for GC Notify Team
5 stars 0 forks source link

Investigate how to snooze repetitive alarms #1566

Open jimleroyer opened 4 months ago

jimleroyer commented 4 months ago

Description

As an ops lead or incident commander, I need the ability to snooze specific alerts so that I can ignore these for a given amount of time.

WHY are we building?

There are times when it is beneficial to ignore specific alarms. For examples, receiving numerous urgent messages through our contact form from the same source, which might be unimportant and outside of business hours, or for a technical error that we know is inconsequential for a few hours and can wait for the business hours to get fixed.

WHAT are we building?

Lead an investigation on solutions to ignore or snooze repetitive alarms. This should come in a written format in a document of your choice (Google doc for example). Describe current alarms architecture with different sources. Describe solutions and alternatives, feasibility, changes and potential cost.

VALUE created by our solution

Brainstorm different ways to implement and adopt a snooze solution to make support life easier. There is a also a cost argument for snoozing alarms that should get ignored, as a call on support guarantees a minimal amount of hours to pay for the employer. Avoiding unnecessary calls would avoid unnecessary cost.

Documentation and Artifacts

Possibly a Google document. Something collaborative and that can get eyes 👀 for reviews.

Acceptance Criteria

QA Steps

foudamo commented 4 months ago

Adding notes from On-call Training Scenario #16.5

Ideas: