Define Monitor Tagging Standards for Datadog

[mchelen-gov]

Problem Statement

For monitors and alerts to be managed successfully across OCTO and VA, it is necessary that they are associated with the correct teams, systems, and environments. How can we standardize our tagging approach so that monitors are managed

Desired Outcome

• OCTO teams have clear guidelines on how to tag their resources in Datadog
• Ownership and point of contact is available for all monitor for cost usage tracking and issue triage
• Quickly and easily find relevant dashboards and monitors when responding to an incident
• Allow bulk management of resources, such as muting all monitors related to a system undergoing maintenance

Acceptance Criteria

• [x] Draft guidelines are created
• [x] Draft is shared with technical SMEs for feedback
• [x] Feedback is incorporated into final version
• [x] Documentation published on Platform website
• [x] Guidelines are applied as monitor tag policies

Involved

@mchelen-gov @va-albers @acrollet

Tasks

• [ ] Draft section: https://vfs.atlassian.net/wiki/spaces/~642825999/pages/2861728279/Monitoring+and+Alerting+Best+Practices#Tagging-Standards
• [ ] Review ECC guidance https://dvagov.sharepoint.com/sites/OITECOECCCOM/Monitoring%20Standards/Forms/AllItems.aspx?id=%2Fsites%2FOITECOECCCOM%2FMonitoring%20Standards%2FMonitoring%20Tool%20Standard%20%2D%20DataDog%2Epdf&parent=%2Fsites%2FOITECOECCCOM%2FMonitoring%20Standards&isSPOFile=1
• [ ] Align OCTO and ECC best practices
• [ ] Roll out initial monitor tagging policy in Datadog

[CoryTrimmUSDS]

@mheadd - is this also considered done? I was looking at #70830 and this issue this morning.

[mheadd]

@va-albers @acrollet Do either of you know the status of this one? I don't see any of the items in the Tasks section above checked, but that may not reflect the state of work on this item.

[acrollet]

It looks like @AparnaNittalaUSDS and @va-albers are looking at this per https://dsva.slack.com/archives/C063TF8RJS2/p1710859336524949?thread_ts=1709649596.197419&cid=C063TF8RJS2

[mchelen-gov]

This was completed, guidance is published here: https://docs.datadoghq.com/monitors/settings/

Admins can edit the enforced config in Datadog "Monitor Tagging Policy" https://docs.datadoghq.com/monitors/settings/

Note that this issue is for one piece of broader monitoring best practice guidelines which have other sections still WIP (how to write good alerts, how to manage signal to noise, etc). @va-albers and @BillChapmanUSDS should have all the Confluence links for those WIP docs.