[DataDog] Reduce noise for Traffic anomaly monitors

department-of-veterans-affairs / va.gov-team

Public resources for building on and in support of VA.gov. Visit complete Knowledge Hub:

https://depo-platform-documentation.scrollhelp.site/index.html

278 stars 195 forks source link

[DataDog] Reduce noise for Traffic anomaly monitors #84484

Open jerekshoe opened 1 month ago

jerekshoe commented 1 month ago

Description

The traffic anomaly monitors go off quite frequently (multiple times a week currently) and since those are watchtower monitors we have been asked to find ways to reduce the noise of these monitors.

Engineering notes

The suggested approach is to utilize a Composite monitor / monitors, mostly likely by mixing in a monitor that goes off when traffic to the CST drops below some positive threshold value

Tasks

[ ] Determine how we want to implement this

jerekshoe commented 1 month ago

This ticket should be fleshed out more in the next Engineering refinement