Monitoring using Prometheus

[fischerman]

This is related to monitoring Consul Template. The discussion in #570 has converged to sending health data to Consul (issue has been stale since early 2018). I'm proposing to expose metrics using the Prometheus (soon OpenMetrics format)

Here are my thoughts:

• Works well in Kubernetes settings. Prometheus is most common monitoring solution right now.
• We are using Consul Template as a sidecar. Hence, there is no port conflict (which is one of the concerns by @sethvargo in #570). The http server could be disabled by default, to prevent breaking changes.
• Consul Template can be used just for Vault, in which case there is no Consul to send data to.
• Other HashiCorp products adapt Prometheus as well (e.g. see https://github.com/hashicorp/vault/pull/5308)

[eikenb]

Hey @fischerman, thanks for submitting this.

Going to pre-thank you for your patience as we make time to get to this issue. We will be super busy with an internal project for the next few weeks and won't be able to look at this until after that.

On the up side though the team for these projects is growing, currently there are 2 of us and we might be getting a 3rd person. So I'm hoping we will be able to not only pay more attention to incoming issues/enhancements but really catch up with all the backlog. Again let me convey my appreciation for taking the time to help and for your patience while we get caught up with everything.

[eikenb]

Moving to go-metrics. See #1395.