Consul emits extra dummy metrics to Prometheus which always have zero values

[maxb]

Example

# HELP consul_consul_state_services Measures the current number of unique services registered with Consul, based on service name. It is only emitted by Consul servers. Added in v1.9.0.
# TYPE consul_consul_state_services gauge
consul_consul_state_services 0
consul_consul_state_services{datacenter="dc1"} 1

Explanation

The real values for consul_consul_state_services are emitted with a datacenter label.

However there is an extra metric with an empty label set, which always has value zero, and is never updated.

It would be nice to remove this metric, however the underlying go-metrics library in use, does not allow this, without also sacrificing the help message. I have opened https://github.com/armon/go-metrics/issues/136 about this.

The primary purpose of opening this Consul issue in the short term, is to:

• Record that this quirk exists and IMO would be worth documenting in the "Telemetry" documentation
• Provide an URL that can be referenced in comments in the source code, to provide a central explanation why various metrics are defined but never used.

[Amier3]

Thanks for making this issue as well, this is definitely worth documenting. This'll require some discussion along with #13494, but I wanted comment on each individual issue to acknowledge that we do see these and will talk about a way to address them ( possibly together ).

[jakubgs]

Just noticed this issue when I tried to enable metrics for my cluster. The result is that I see zeroes in the queries: