CoreServices_CollectdMlabMissing - Githubissues

m-lab / scraper

Scrape experiment data off of MLab nodes and upload it to Google Cloud Storage

Apache License 2.0

5 stars 5 forks source link

CoreServices_CollectdMlabMissing #297

Closed measurementlab closed 6 years ago

measurementlab commented 6 years ago

Alertmanager URL: https://mlab:YOztKFSKnRMz2GN1qFPueAku9WhmDYV2@alertmanager.mlab-oti.measurementlab.net

firing https://prometheus.mlab-oti.measurementlab.net/graph?g0.expr=up%7Bservice%3D%22nodeexporter%22%7D+%3D%3D+1+unless+ON%28machine%29+collectd_mlab_success&g0.tab=1

Labels:
- alertname = CoreServices_CollectdMlabMissing
- instance = mlab1.tyo02.measurement-lab.org:9100
- job = legacy-targets
- machine = mlab1.tyo02.measurement-lab.org
- service = nodeexporter
- severity = ticket
Annotations:
- hints = The collectd-mlab service runs in the mlab_utility slice. Try running the ansible/disco/update-mlab-utility.yaml Ansible playbook in the mlabops repository to configure collectd-mlab. Login to the node and run the check script manually to see what the specific error is (/usr/lib/nagios/plugins/check_collectd_mlab.py).
- summary = A collectd-mlab service metric is missing.

TODO: add graph url from annotations.

stephen-soltesz commented 6 years ago

This is due to enabling partial monitoring of the TYO VMs running in GCE without all core services.

Added silence for 90d. https://alertmanager.mlab-oti.measurementlab.net/#/silences/62333066-cd6c-47dd-9929-4c70d65a415b

Updated https://github.com/m-lab/operator/issues/154 to resolve issues around tyo monitoring.