Audit and improve alerts - Githubissues

department-of-veterans-affairs / va.gov-team

Public resources for building on and in support of VA.gov. Visit complete Knowledge Hub:

https://depo-platform-documentation.scrollhelp.site/index.html

283 stars 204 forks source link

Audit and improve alerts #97029

Open comaurice opened 1 day ago

comaurice commented 1 day ago

Value Statement

As a Dev

I want to streamline all the DR alerts that we currently generate

So that I have clarity on the important slack alerts

Background Context

We have a lot of alert monitors set up in DataDog, and also alerts that come through a few other channels. Can we consolidate, or reduce noise?

Outcome, Success Measure, KPI(S), and Tracking Link

Every alert that appears in our Slack channel requires action or awareness within 24 hours

Acceptance Criteria

[ ] Updated playbooks for alerts
[ ] Clear action for each alert or warning

Designs and Build Notes

Add here

Tasks

[ ] List of alerts, including source and delivery mechanism
[ ] Can we use our new metrics to get better alerts less often?

Definition of Ready

[ ] Clear value description
[ ] Testable acceptance criteria
[ ] Dependencies and blockers linked

Definition of Done

[ ] Meets acceptance criteria
[ ] New playbooks shared
[ ] Team approved production verification process
[ ] Reviewed and approved by product and/or design

kayline commented 1 day ago

All our current monitors