LameDuckMetricMissingForNode - Githubissues

m-lab / scraper

Scrape experiment data off of MLab nodes and upload it to Google Cloud Storage

Apache License 2.0

5 stars 5 forks source link

LameDuckMetricMissingForNode #298

Closed measurementlab closed 6 years ago

measurementlab commented 6 years ago

Alertmanager URL: https://alertmanager.mlab-oti.measurementlab.net

firing https://prometheus.mlab-oti.measurementlab.net/graph?g0.expr=count%28up%7Bservice%3D%22nodeexporter%22%7D+%3D%3D+1+unless+ON%28machine%29+lame_duck_node%29+%3E+0&g0.tab=1

Labels:
- alertname = LameDuckMetricMissingForNode
- severity = ticket
Annotations:
- hints = Check /var/spool/node_exporter/ on the node to see if the file lame_duck.prom is missing. If it is, use the mlabops Ansible lame-duck.yaml playbook to restore it.
- summary = Some number of nodes are missing lame-duck status metrics.

TODO: add graph url from annotations.

stephen-soltesz commented 6 years ago

Same root cause as https://github.com/m-lab/scraper/issues/297 except this alert cannot be silenced yet.

Opened issue https://github.com/m-lab/prometheus-support/issues/227 to resolve the issue with this alert.

Leaving issue open so a new one is not created.