ParserDailyVolumeTooLow

[measurementlab]

Alertmanager URL: http://status.mlab-oti.measurementlab.net:9093

• firing http://status.mlab-oti.measurementlab.net:9090/graph?g0.expr=sum%28increase%28etl_test_count%7Bservice%21~%22.%2Abatch.%2A%22%7D%5B1d%5D%29%29+BY+%28service%29+%3C+%280.7+%2A+quantile%280.5%2C+label_replace%28sum%28increase%28etl_test_count%7Bservice%21~%22.%2Abatch.%2A%22%7D%5B1d%5D+OFFSET+1d%29%29+BY+%28service%29%2C+%22delay%22%2C+%221d%22%2C+%22%22%2C+%22.%2A%22%29+or+label_replace%28sum%28increase%28etl_test_count%7Bservice%21~%22.%2Abatch.%2A%22%7D%5B1d%5D+OFFSET+3d%29%29+BY+%28service%29%2C+%22delay%22%2C+%223d%22%2C+%22%22%2C+%22.%2A%22%29+or+label_replace%28sum%28increase%28etl_test_count%7Bservice%21~%22.%2Abatch.%2A%22%7D%5B1d%5D+OFFSET+5d%29%29+BY+%28service%29%2C+%22delay%22%2C+%225d%22%2C+%22%22%2C+%22.%2A%22%29+or+label_replace%28sum%28increase%28etl_test_count%7Bservice%21~%22.%2Abatch.%2A%22%7D%5B1d%5D+OFFSET+1w%29%29+BY+%28service%29%2C+%22delay%22%2C+%227d%22%2C+%22%22%2C+%22.%2A%22%29+or+label_replace%28label_replace%28vector%280%29%2C+%22delay%22%2C+%22c1%22%2C+%22%22%2C+%22.%2A%22%29%2C+%22service%22%2C+%22etl-ndt-parser%22%2C+%22%22%2C+%22.%2A%22%29+or+label_replace%28label_replace%28vector%280%29%2C+%22delay%22%2C+%22c2%22%2C+%22%22%2C+%22.%2A%22%29%2C+%22service%22%2C+%22etl-ndt-parser%22%2C+%22%22%2C+%22.%2A%22%29+or+label_replace%28label_replace%28vector%280%29%2C+%22delay%22%2C+%22c1%22%2C+%22%22%2C+%22.%2A%22%29%2C+%22service%22%2C+%22etl-sidestream-parser%22%2C+%22%22%2C+%22.%2A%22%29+or+label_replace%28label_replace%28vector%280%29%2C+%22delay%22%2C+%22c2%22%2C+%22%22%2C+%22.%2A%22%29%2C+%22service%22%2C+%22etl-sidestream-parser%22%2C+%22%22%2C+%22.%2A%22%29+or+label_replace%28label_replace%28vector%280%29%2C+%22delay%22%2C+%22c1%22%2C+%22%22%2C+%22.%2A%22%29%2C+%22service%22%2C+%22etl-traceroute-parser%22%2C+%22%22%2C+%22.%2A%22%29+or+label_replace%28label_replace%28vector%280%29%2C+%22delay%22%2C+%22c2%22%2C+%22%22%2C+%22.%2A%22%29%2C+%22service%22%2C+%22etl-traceroute-parser%22%2C+%22%22%2C+%22.%2A%22%29%29+BY+%28service%29%29&g0.tab=1

  Labels:

  • alertname = ParserDailyVolumeTooLow
  • service = etl-traceroute-parser
  • severity = page

  Annotations:

  • hints = Are machines online? Is data being collected? Is the parser working?
  • summary = Today's test volume is less than 70% of nominal daily volume.
  • url = http://status.mlab-oti.measurementlab.net:3000/dashboard/file/Alert_ParserDailyVolumeTooLow.json

TODO: add graph url from annotations.

[gfr10598]

This will be a recurring alert over the next few days. Following the PT panic bug, we had to catch up on several days of data, which produced a large spike in the history.

[stephen-soltesz]

Added 1w silence or disco.