ClusterDown

[measurementlab]

Alertmanager URL: http://status.mlab-oti.measurementlab.net:9093

• firing http://status.mlab-oti.measurementlab.net:9090/graph?g0.expr=up%7Bjob%3D%22federation-targets%22%7D+%3D%3D+0&g0.tab=1

  Labels:

  • alertname = ClusterDown
  • cluster = scraper-cluster
  • instance = 35.188.22.107:9090
  • job = federation-targets
  • service = prometheus-public-service
  • severity = page
  • zone = us-central1-a

  Annotations:

  • description = 35.188.22.107:9090 of job federation-targets has been down for more than 2 minutes.
  • summary = Instance 35.188.22.107:9090 down

TODO: add graph url from annotations.

[stephen-soltesz]

This was due to a cluster node auto-upgrade! 1.7.11 -> 1.8.6 -- the sensitivity of ClusterDown alert should be adjusted. e.g. from 3m to 10m to prevent the false positive.

Created: https://github.com/m-lab/prometheus-support/pull/180 to fix.

http://35.188.22.107:9090/graph?g0.range_input=6h&g0.end_input=2018-01-25+14%3A40&g0.step_input=120&g0.stacked=0&g0.expr=rate(process_cpu_seconds_total%7Bcontainer%3D%22prometheus%22%7D%5B5m%5D)&g0.tab=0&g1.range_input=6h&g1.end_input=2018-01-25+14%3A40&g1.expr=kube_node_info%7Bcluster%3D%22scraper-cluster-prometheus-pool%22%2C+node%3D~%22gke-scraper-cluster-prometheus-pool.*%22%7D%09&g1.tab=0