Better consul monitoring!

[tomwilkie]

For instance, we should alert when consul fails to elect a master.

More idea (for etc, but should be transferable):

https://coreos.com/blog/developing-prometheus-alerts-for-etcd.html

[tomwilkie]

Related:

• https://engineering.datadoghq.com/consul-at-datadog/index.html
• https://github.com/prometheus/consul_exporter

[jml]

Lack of this made weaveworks/service-conf#214 more severe & harder to debug.

[tomwilkie]

Need to deploy the extra consul exporter, but with prometheus can only scrap one port per service right now. This will be fixed by fabian's prometheus k8s disco rewrite, which will land soon.

[tomwilkie]

https://www.consul.io/docs/agent/telemetry.html