m-lab / prometheus-support

Prometheus configuration for M-Lab running on GKE
Apache License 2.0
19 stars 11 forks source link

Reduce sensitivity of Parser alert #1045

Closed cristinaleonr closed 3 months ago

cristinaleonr commented 3 months ago

This PR reduces the sensitivity of the ParserFailureRateTooHighOrMissing alert. The alert recently fired%20by%20(table)%20%2F%0A%20%20%20%20%20%20sum(rate(etl_task_total%7Bcluster%3D%22data-pipeline%22%7D%5B1h%5D))%20by%20(table))%20%3E%200.01%0A%20%20%20%20%20%20OR%20absent(etl_task_total%7Bcluster%3D%22data-pipeline%22%7D)&g0.tab=0&g0.display_mode=lines&g0.show_exemplars=0&g0.range_input=12h) because of an ephemeral event that lasted ~20 mins (could be caused because of a pod restart).


This change is Reviewable