ParserDailyVolumeTooLow

[measurementlab]

Alertmanager URL: https://mlab:YOztKFSKnRMz2GN1qFPueAku9WhmDYV2@alertmanager.mlab-oti.measurementlab.net

• firing https://prometheus.mlab-oti.measurementlab.net/graph?g0.expr=sum%28increase%28etl_test_count%7Bservice%21~%22.%2Abatch.%2A%22%2Cstatus%3D%22ok%22%7D%5B1d%5D%29%29+BY+%28service%29+%3C+%280.7+%2A+quantile%280.5%2C+label_replace%28sum%28increase%28etl_test_count%7Bservice%21~%22.%2Abatch.%2A%22%2Cstatus%3D%22ok%22%7D%5B1d%5D+OFFSET+1d%29%29+BY+%28service%29%2C+%22delay%22%2C+%221d%22%2C+%22%22%2C+%22.%2A%22%29+or+label_replace%28sum%28increase%28etl_test_count%7Bservice%21~%22.%2Abatch.%2A%22%2Cstatus%3D%22ok%22%7D%5B1d%5D+OFFSET+3d%29%29+BY+%28service%29%2C+%22delay%22%2C+%223d%22%2C+%22%22%2C+%22.%2A%22%29+or+label_replace%28sum%28increase%28etl_test_count%7Bservice%21~%22.%2Abatch.%2A%22%2Cstatus%3D%22ok%22%7D%5B1d%5D+OFFSET+5d%29%29+BY+%28service%29%2C+%22delay%22%2C+%225d%22%2C+%22%22%2C+%22.%2A%22%29+or+label_replace%28sum%28increase%28etl_test_count%7Bservice%21~%22.%2Abatch.%2A%22%2Cstatus%3D%22ok%22%7D%5B1d%5D+OFFSET+1w%29%29+BY+%28service%29%2C+%22delay%22%2C+%227d%22%2C+%22%22%2C+%22.%2A%22%29+or+label_replace%28label_replace%28vector%280%29%2C+%22delay%22%2C+%22c1%22%2C+%22%22%2C+%22.%2A%22%29%2C+%22service%22%2C+%22etl-disco-parser%22%2C+%22%22%2C+%22.%2A%22%29+or+label_replace%28label_replace%28vector%280%29%2C+%22delay%22%2C+%22c2%22%2C+%22%22%2C+%22.%2A%22%29%2C+%22service%22%2C+%22etl-disco-parser%22%2C+%22%22%2C+%22.%2A%22%29+or+label_replace%28label_replace%28vector%280%29%2C+%22delay%22%2C+%22c1%22%2C+%22%22%2C+%22.%2A%22%29%2C+%22service%22%2C+%22etl-ndt-parser%22%2C+%22%22%2C+%22.%2A%22%29+or+label_replace%28label_replace%28vector%280%29%2C+%22delay%22%2C+%22c2%22%2C+%22%22%2C+%22.%2A%22%29%2C+%22service%22%2C+%22etl-ndt-parser%22%2C+%22%22%2C+%22.%2A%22%29+or+label_replace%28label_replace%28vector%280%29%2C+%22delay%22%2C+%22c1%22%2C+%22%22%2C+%22.%2A%22%29%2C+%22service%22%2C+%22etl-sidestream-parser%22%2C+%22%22%2C+%22.%2A%22%29+or+label_replace%28label_replace%28vector%280%29%2C+%22delay%22%2C+%22c2%22%2C+%22%22%2C+%22.%2A%22%29%2C+%22service%22%2C+%22etl-sidestream-parser%22%2C+%22%22%2C+%22.%2A%22%29+or+label_replace%28label_replace%28vector%280%29%2C+%22delay%22%2C+%22c1%22%2C+%22%22%2C+%22.%2A%22%29%2C+%22service%22%2C+%22etl-traceroute-parser%22%2C+%22%22%2C+%22.%2A%22%29+or+label_replace%28label_replace%28vector%280%29%2C+%22delay%22%2C+%22c2%22%2C+%22%22%2C+%22.%2A%22%29%2C+%22service%22%2C+%22etl-traceroute-parser%22%2C+%22%22%2C+%22.%2A%22%29%29+BY+%28service%29%29&g0.tab=1

  Labels:

  • alertname = ParserDailyVolumeTooLow
  • service = etl-ndt-parser
  • severity = page

  Annotations:

  • hints = Are machines online? Is data being collected? Is the parser working?
  • summary = Today's test volume is less than 70% of nominal daily volume.
  • url = http://status.mlab-oti.measurementlab.net:3000/dashboard/file/Alert_ParserDailyVolumeTooLow.json

TODO: add graph url from annotations.

[pboothe]

This alert was caused by a push of bad cloud function code. Things have been fixed with https://github.com/m-lab/etl/pull/506 and https://github.com/m-lab/etl/pull/507

The alert will keep firing for the next ~24 hours, though. I have silenced this alert for the next 24 hours.

[pboothe]

If it fires again, then we will want to be notified about it, because it will mean that I failed to fix it. Closing this issue.