discordianfish / nginx_exporter

Simple server that scrapes nginx stats and exports them via HTTP for Prometheus consumption
MIT License
160 stars 76 forks source link

nginx_exporter failing with "panic: inconsistent label cardinality" #9

Closed jforest closed 8 years ago

jforest commented 8 years ago

My go version: nginx_exporter@test1:~/nginx_exporter_repo$ go version go version go1.6.2 linux/amd64

My linux version: nginx_exporter@test1:~/nginx_exporter_repo$ cat /etc/issue Debian GNU/Linux 8 \n \l

My error:

nginx_exporter@test1:~/nginx_exporter_repo$ ./nginx_exporter -log.level debug -nginx.scrape_uri http://localhost:19114/nginx_status -telemetry.address :9114
INFO[0000] Starting Server: :9114                        file=nginx_exporter.go line=174
panic: inconsistent label cardinality

goroutine 29 [running]:
panic(0x7f63e0, 0xc820011580)
    /usr/lib/go-1.6/src/runtime/panic.go:481 +0x3e6
github.com/prometheus/client_golang/prometheus.MustNewConstMetric(0xc8200e8070, 0x1, 0x4049000000000000, 0xc820174040, 0x1, 0x1, 0x0, 0x0)
    /home/nginx_exporter/gocode/src/github.com/prometheus/client_golang/prometheus/value.go:172 +0xd0
main.(*Exporter).collect(0xc8200ce190, 0xc8200f0240, 0x0, 0x0)
    /home/nginx_exporter/nginx_exporter_repo/nginx_exporter.go:119 +0x998
main.(*Exporter).Collect(0xc8200ce190, 0xc8200f0240)
    /home/nginx_exporter/nginx_exporter_repo/nginx_exporter.go:160 +0xa1
github.com/prometheus/client_golang/prometheus.(*Registry).Gather.func2(0xc8200d64e0, 0xc8200f0240, 0x7f945fab5298, 0xc8200ce190)
    /home/nginx_exporter/gocode/src/github.com/prometheus/client_golang/prometheus/registry.go:382 +0x58
created by github.com/prometheus/client_golang/prometheus.(*Registry).Gather
    /home/nginx_exporter/gocode/src/github.com/prometheus/client_golang/prometheus/registry.go:383 +0x360

Seems related to the latest merge, but I'm not a go expert :/

jforest commented 8 years ago

oh, and the endpoint nginx_exporter should be getting data from:

nginx_exporter@test1:~/nginx_exporter_repo$ curl http://localhost:19114/nginx_status
Active connections: 1
server accepts handled requests
 55 55 82
Reading: 0 Writing: 1 Waiting: 0
jforest commented 8 years ago

Ok, so I finally got this (sort of) figured out. I think the updated way of setting the processedConnections in the NewExporter function isn't quite right. I forked the prometheus/client_golang to print out where it was expecting the length of the labelValues array to be equal to the length of the desc.variableLabels array. With the method from the last pull request, it was setting labelValues to an array 3 elements long. ["accepted", "handled", "any"] But, it was only passing an array 1 element long in desc.variableLabels ["accepted"] or ["handled"] or ["any"]

Changing to send 3 values ends up with metrics like:

nginx_connections_processed_total{accepted="accepted",any="acc",handled="accep"} 82
nginx_connections_processed_total{accepted="any",any="blah3",handled="blah2"} 109
nginx_connections_processed_total{accepted="handled",any="blah",handled="blah"} 82

and with my branch that I'll submit a pull request for:

nginx_connections_processed_total{stage="accepted"} 86
nginx_connections_processed_total{stage="any"} 133
nginx_connections_processed_total{stage="handled"} 86
discordianfish commented 8 years ago

Assuming this can be closed.

tamsky commented 8 years ago

Thanks for updating your dockerhub* image so quickly!

discordianfish commented 8 years ago

@tamsky That should happen automatically :)