search

chop-dbhi / prometheus-sql

Service that exposes Prometheus metrics for a SQL result set.
BSD 2-Clause "Simplified" License
202 stars 55 forks source link

two queries with same name but different labels crashes the exporter #31

Closed yosefy closed 7 years ago

yosefy commented 7 years ago

=================================== log:

Registering metric query with facets {}

panic: a previously registered descriptor with the same fully-qualified name as Desc{fqName: "query_result_query", help: "Result of an SQL query", constLabels: {}, variableLabels: []} has different label names or a different help string

goroutine 11 [running]: github.com/prometheus/client_golang/prometheus.(Registry).MustRegister(0xc420016900, 0xc4201a8200, 0x1, 0x1) /go/src/github.com/prometheus/client_golang/prometheus/registry.go:353 +0x92 github.com/prometheus/client_golang/prometheus.MustRegister(0xc4201a8200, 0x1, 0x1) /go/src/github.com/prometheus/client_golang/prometheus/registry.go:152 +0x53 main.(QueryResult).registerMetric(0xc4200f9250, 0xc42020ff20, 0xc420045b58, 0x0) /go/pipelines/prometheus-sql/set.go:47 +0x666 main.(QueryResult).SetMetrics(0xc4200f9250, 0xc420045d38, 0x1, 0x1, 0xc42009fe60, 0xc42009fe60, 0x946c30) /go/pipelines/prometheus-sql/set.go:101 +0x425 main.(Worker).SetMetrics(0xc4200ebe30, 0xc420045d38, 0x1, 0x1) /go/pipelines/prometheus-sql/worker.go:40 +0x51 main.(Worker).Fetch(0xc4200ebe30, 0x7ffd9443cea5, 0x14, 0x0, 0x0, 0x0, 0x0, 0x0) /go/pipelines/prometheus-sql/worker.go:90 +0x69e main.(Worker).Start.func1() /go/pipelines/prometheus-sql/worker.go:124 +0x45 main.(*Worker).Start(0xc4200ebe30, 0x7ffd9443cea5, 0x14) /go/pipelines/prometheus-sql/worker.go:143 +0x182 created by main.main /go/pipelines/prometheus-sql/main.go:91 +0x77d

yosefy commented 7 years ago

again for no reason this time

this happened to me with python client and was caused by HELP strings added in wrong place

2017-08-21T06:30:47.352747669Z Registering metric XXX with facets {} 2017-08-21T06:30:47.354999259Z panic: a previously registered descriptor with the same fully-qualified name as Desc{fqName: "query_result_XXX", help: "Result of an SQL query", constLabels: {}, variableLabels: []} has different label names or a different help string 2017-08-21T06:30:47.355629119Z 2017-08-21T06:30:47.355638236Z goroutine 13 [running]: 2017-08-21T06:30:47.355642938Z github.com/prometheus/client_golang/prometheus.(Registry).MustRegister(0xc420016900, 0xc4200f58f0, 0x1, 0x1) 2017-08-21T06:30:47.355892146Z /go/src/github.com/prometheus/client_golang/prometheus/registry.go:353 +0x92 2017-08-21T06:30:47.355904551Z github.com/prometheus/client_golang/prometheus.MustRegister(0xc4200f58f0, 0x1, 0x1) 2017-08-21T06:30:47.355908949Z /go/src/github.com/prometheus/client_golang/prometheus/registry.go:152 +0x53 2017-08-21T06:30:47.356860677Z main.(QueryResult).registerMetric(0xc4200f54e0, 0xc4200e78c0, 0xc420043b58, 0x0) 2017-08-21T06:30:47.356875792Z /go/pipelines/prometheus-sql/set.go:47 +0x666 2017-08-21T06:30:47.356879286Z main.(QueryResult).SetMetrics(0xc4200f54e0, 0xc420043d38, 0x1, 0x1, 0xc4201627e0, 0xc4201627e0, 0x946c30) 2017-08-21T06:30:47.356882021Z /go/pipelines/prometheus-sql/set.go:101 +0x425 2017-08-21T06:30:47.356884474Z main.(Worker).SetMetrics(0xc4200ddf10, 0xc420043d38, 0x1, 0x1) 2017-08-21T06:30:47.356886951Z /go/pipelines/prometheus-sql/worker.go:40 +0x51 2017-08-21T06:30:47.356889344Z main.(Worker).Fetch(0xc4200ddf10, 0x7fffc4fcbea5, 0x14, 0x0, 0x0, 0x0, 0x0, 0x0) 2017-08-21T06:30:47.356891741Z /go/pipelines/prometheus-sql/worker.go:90 +0x69e 2017-08-21T06:30:47.356894097Z main.(Worker).Start.func1() 2017-08-21T06:30:47.356896373Z /go/pipelines/prometheus-sql/worker.go:124 +0x45 2017-08-21T06:30:47.356898791Z main.(*Worker).Start(0xc4200ddf10, 0x7fffc4fcbea5, 0x14) 2017-08-21T06:30:47.356900964Z /go/pipelines/prometheus-sql/worker.go:143 +0x182 2017-08-21T06:30:47.356903333Z created by main.main 2017-08-21T06:30:47.356905573Z /go/pipelines/prometheus-sql/main.go:91 +0x77d

yosefy commented 7 years ago

@haxorof , can you help with that? as for now it became not usable i can provide all the info needed thanks

haxorof commented 7 years ago

@yosefy: It is not clear to me what output you want to see in the metrics endpoint. My feeling is that when you specify two queries with the same name the registration in the Prometheus client library will not be correct. This might be the reason for the crashes you see but I have not verified this is true.

Can you please show how you expect the output to look like in the metrics endpoint. Something similar to this?

query_result_query{type="MSSQL", client="Client1"} 1
query_result_query{type="MSSQL", client="Client2"} 10

If it is something similar to the above you want then can you provide more details of the actual SQL statement and the table? Maybe there is another way to solve it by changing the SQL statement.

Cheers!

yosefy commented 7 years ago

yes this is exactly the situation it crashes i am almost sure its because of HELP stuff in wrong loop or something

query doesnt matter as it crashed for multiple queries for me but for example

if i change

to

all good

in logs: panic: a previously registered descriptor with the same fully-qualified name as Desc{fqName: "query_result_XXX", help: "Result of an SQL query", constLabels: {}, variableLabels: []} has different label names or a different help string

thanks

haxorof commented 7 years ago

Then it is what I thought that it is how prometheus-sql register metrics into the prometheus client library. It will complain about this help string when it is registered incorrectly. So basically I can say prometheus-sql does not support to have two queries with the same name. I would say this mechanism will most likely not be changed because that will not be an easy change at all and keep compatibility etc.

@yosefy About your query, is it that GROUP_ID basically defines which client it is? If that is the case could you not try to use some control-flow function like CASE to get only one query? https://dev.mysql.com/doc/refman/5.7/en/control-flow-functions.html

Cheers!

yosefy commented 7 years ago

well then it is not usable for us. which is very sad :)

the situation is we have some metric and then clients as a label

then we may make graphs in grafana

if i have to call it different name every time, graph can't be done

thanks

haxorof commented 7 years ago

I see now that I did not understand your use case correctly in my last message. Sadly how prometheus-sql is written today I agree that you can't use it for this type of use case you are looking for.

Hope you will find some other way to solve what you want to achieve!

haxorof commented 7 years ago

Maybe a more clumsy option would be to run two instances of prometheus-sql in parallel and then let each one run one of the query. Then you need to specify two targets in Prometheus which then separate the "clients". Would something similar to that work @yosefy?

yosefy commented 7 years ago

not an option as we have same metric for lots of clients

nevermind

thanks Bjorn

On Sun, Sep 10, 2017 at 11:11 AM, Björn Oscarsson notifications@github.com wrote:

Maybe a more clumsy option would be to run two instances of prometheus-sql in parallel and then let each one run one of the query. Then you need to specify two targets in Prometheus which then separate the "clients". Would something similar to that work @yosefy https://github.com/yosefy?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/chop-dbhi/prometheus-sql/issues/31#issuecomment-328326940, or mute the thread https://github.com/notifications/unsubscribe-auth/AMVy7Qa7BrP-7iMeRv718Qs_Nic8gjoMks5sg5mdgaJpZM4O8hjk .