create alert for dramatic daily data volumn change per test - Githubissues

m-lab / prometheus-support

Prometheus configuration for M-Lab running on GKE

Apache License 2.0

19 stars 11 forks source link

create alert for dramatic daily data volumn change per test #580

Open yachang opened 4 years ago

yachang commented 4 years ago

https://grafana.mlab-oti.measurementlab.net/d/WnaxPZJZz/alert-platformcluster_pusherdailydatavolumetoolow?orgId=1&from=now-7d&to=now&var-project=oti&var-cluster=Platform%20Cluster%20(mlab-oti)&var-federation=Prometheus%20(mlab-oti)

Based on the metrics above, if data volumn drop more than ~20% compared to past 10 days average, send an alert for the possible screw-up.

stephen-soltesz commented 4 years ago

We have this alert for data volume. The dashboard linked above corresponds to this alert - https://github.com/m-lab/prometheus-support/blob/master/config/federation/prometheus/alerts.yml#L572