Group/Tag Code Yellow Tier 1 Features

[mheadd]

Description

Group/tag the code yellow tier 1 feature monitors so we can filter appropriately. Datadog's Triggered Monitors section seems to be what we're looking for (for example, Filter by Triggered Within:24hr).

Discussion here.

Ideas

• Tier 1 feature monitors are tagged as "P1 (Critical)" Priority
• Tier 1 feature monitors have a new and special tag: "Tier 1", "Code Yellow Monitoring and Observability", "Watch Officer", ___?
• We create a "Watch Officer" Team and then tag all tier 1 feature monitors to the Watch Officer Team
• Other alternatives

Definition of done

• [x] Decide on a tagging/grouping approach
• [ ] Tag Tier 1 feature monitors
• [ ] Separate tag for tier-1 service monitors that should be visible in watch-officer dashboard
• [ ] Organize tagged monitors for Watch Officer Team
• [ ] Remove from watchtower dashboard monitors without "watchtower" tag

  ### Tasks

[zachgoldfine44]

Suggestion: custom tag: "watch officer".

Steve will put on all code yellow core requirement monitors

[zachgoldfine44]

"Watchtower" tag added to all core requirement monitors. Verified that the watchtower dashboard can be filtered so that only monitors with "watchtower" tag show.

Next up: Steve to tag all monitors from spreadsheet with new watchtower tag.

[va-albers]

All monitors in the spreadsheet (including ones added today) are tagged with "watchtower", and the dashboard is exclusively filtering on this tag now.