search

Showmax / prometheus-pusher

Scrapes local metric from local exporters and pushes them towards Prometheus pushgateway
http://tech.showmax.com
Other
36 stars 16 forks source link

Prometheus-pushgateway has lots of warning messges #12

Closed harighub closed 6 years ago

harighub commented 7 years ago

Configuration

We have deployed a pod on k8s where prometheus and prometheus-pushgateway is configured . In another pod prometheus pusher (https://github.com/Showmax/prometheus-pusher) is sending metrics to prometheus-pushgateway.

Observation:

prometheus-pushgateway is overwhelming with messages like below:

time="2017-08-24T15:56:51Z" level=warning msg="Metric families 'name:\"go_memstats_sys_bytes\" help:\"Number of bytes obtained from system.\" type:GAUGE metric:<label:<name:\"instance\" value:\"host-level-ds-rh9wk\" > label:<name:\"job\" value:\"target_9126\" > gauge:<value:4.9817848e+07 > timestamp_ms:1503590202395 > ' and 'name:\"go_memstats_sys_bytes\" help:\"Number of bytes obtained by system. Sum of all system allocations.\" type:GAUGE metric:<label:<name:\"instance\" value:\"service1-4138284533-4xshv\" > label:<name:\"job\" value:\"target_9102\" > gauge:<value:1.1147512e+07 > timestamp_ms:1503590059331 > metric:<label:<name:\"instance\" value:\"service1-4138284533-cjzpd\" > label:<name:\"job\" value:\"target_9102\" > gauge:<value:1.1147512e+07 > timestamp_ms:1503590059714 > metric:<label:<name:\"instance\" value:\"service1-4138284533-gcqsf\" > label:<name:\"job\" value:\"target_9102\" > gauge:<value:1.1147512e+07 > timestamp_ms:1503590057324 > ' are inconsistent, help and type of the latter will have priority. This is bad. Fix your pushed metrics!" source="diskmetricstore.go:114"

As you can see, the help string is slightly different for the two metrics of the same name. The Prometheus data model requires that metrics of the same name also have the same help string.

To get rid of the warnings, you have to fix your pushers to push the same help string in both cases.

Do we have any pointers?

harighub commented 6 years ago

This doesn't seem to be related to prometheus-pusher. issue was due to duplicate metrics with different help strings in the metrics. closing the issue.