search

mozmeao / snippets-service

Service powering snippets on Firefox's about:home.
https://snippets.mozilla.com
Mozilla Public License 2.0
31 stars 36 forks source link

Create threshold based monitor for Snippet Metrics #816

Closed glogiotatidis closed 6 years ago

glogiotatidis commented 6 years ago

Snippet metrics collection is monitored by a Datadog Anomaly detection monitor.

Since the release of Firefox 63 we don't serve snippets to users due to bug 1503047 (see also #815) but the monitor didn't go off.

Is seems the gradual rollout of Firefox 63, resulted in a steady drop of collected metrics with the rate of drop being within the limits of the anomaly detection algorithm.

image

The monitor was designed to adapt to the overall trend, to prevent false positives during weekends and holiday seasons, and it did act as expected.

In addition to this monitor we need to have another threshold based monitor, with a low trigger value, so we get alerted in situations like this.

glogiotatidis commented 6 years ago

Created new threshold based monitor

https://app.datadoghq.com/monitors/6919608