Exemplar support with prometheus v2.26.0

[bobrik]

Prometheus v2.26 came out with exemplar support:

• https://github.com/prometheus/prometheus/releases/tag/v2.26.0

I can query them when talking directly to Prometheus, but when not when going via promxy (v0.0.70):

404 page not found

It would be great to support exemplars.

cc @prymitive

[PaulFurtado]

It appears @bobrik is getting a 404 error from this endpoint, however, we're actually getting a 500 with a panic on our end:

Error running handler: goroutine 2157390 [running]:
runtime/debug.Stack(0xc00169acc8, 0x2a5b2e0, 0x4c2cca0)
    /usr/local/go/src/runtime/debug/stack.go:24 +0x9f
github.com/jacksontj/promxy/pkg/logging.(*ApacheLoggingHandler).runHandler.func1(0xc00169bad0)
    /usr/share/hubspot/build/workspace/InfraOrch/Docker/promxy/gopath/src/github.com/jacksontj/promxy/pkg/logging/logging.go:139 +0x46
panic(0x2a5b2e0, 0x4c2cca0)
    /usr/local/go/src/runtime/panic.go:965 +0x1b9
github.com/prometheus/prometheus/web/api/v1.(*API).queryExemplars(0xc0005a8900, 0xc001913300, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
    /usr/share/hubspot/build/workspace/InfraOrch/Docker/promxy/gopath/src/github.com/jacksontj/promxy/vendor/github.com/prometheus/prometheus/web/api/v1/api.go:506 +0x8ab
github.com/prometheus/prometheus/web/api/v1.(*API).Register.func1.1(0x380b730, 0xc0016dc9c0, 0xc001913300)
    /usr/share/hubspot/build/workspace/InfraOrch/Docker/promxy/gopath/src/github.com/jacksontj/promxy/vendor/github.com/prometheus/prometheus/web/api/v1/api.go:265 +0xa5
net/http.HandlerFunc.ServeHTTP(0xc00068bb90, 0x380b730, 0xc0016dc9c0, 0xc001913300)
    /usr/local/go/src/net/http/server.go:2049 +0x44
github.com/prometheus/prometheus/util/httputil.CompressionHandler.ServeHTTP(0x37d54a0, 0xc00068bb90, 0x7f034ab839e0, 0xc00558d620, 0xc001913300)
    /usr/share/hubspot/build/workspace/InfraOrch/Docker/promxy/gopath/src/github.com/jacksontj/promxy/vendor/github.com/prometheus/prometheus/util/httputil/compression.go:90 +0x7e
github.com/prometheus/prometheus/web.(*Handler).testReady.func1(0x7f034ab839e0, 0xc00558d620, 0xc001913300)
    /usr/share/hubspot/build/workspace/InfraOrch/Docker/promxy/gopath/src/github.com/jacksontj/promxy/vendor/github.com/prometheus/prometheus/web/web.go:529 +0x55
net/http.HandlerFunc.ServeHTTP(0xc00068bbc0, 0x7f034ab839e0, 0xc00558d620, 0xc001913300)
    /usr/local/go/src/net/http/server.go:2049 +0x44
github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerResponseSize.func1(0x7f034ab839e0, 0xc00558d5f0, 0xc001913300)
    /usr/share/hubspot/build/workspace/InfraOrch/Docker/promxy/gopath/src/github.com/jacksontj/promxy/vendor/github.com/prometheus/client_golang/prometheus/promhttp/instrument_server.go:198 +0xee
net/http.HandlerFunc.ServeHTTP(0xc000bc1aa0, 0x7f034ab839e0, 0xc00558d5f0, 0xc001913300)
    /usr/local/go/src/net/http/server.go:2049 +0x44
github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerDuration.func2(0x7f034ab839e0, 0xc00558d5f0, 0xc001913300)
    /usr/share/hubspot/build/workspace/InfraOrch/Docker/promxy/gopath/src/github.com/jacksontj/promxy/vendor/github.com/prometheus/client_golang/prometheus/promhttp/instrument_server.go:76 +0xb2
net/http.HandlerFunc.ServeHTTP(0xc000bc1c20, 0x7f034ab839e0, 0xc00558d5f0, 0xc001913300)
    /usr/local/go/src/net/http/server.go:2049 +0x44
github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerCounter.func1(0x380b010, 0xc001680f30, 0xc001913300)
    /usr/share/hubspot/build/workspace/InfraOrch/Docker/promxy/gopath/src/github.com/jacksontj/promxy/vendor/github.com/prometheus/client_golang/prometheus/promhttp/instrument_server.go:101 +0xdf
github.com/prometheus/prometheus/web.setPathWithPrefix.func1.1(0x380b010, 0xc001680f30, 0xc001913200)
    /usr/share/hubspot/build/workspace/InfraOrch/Docker/promxy/gopath/src/github.com/jacksontj/promxy/vendor/github.com/prometheus/prometheus/web/web.go:1151 +0x1dc
github.com/prometheus/common/route.(*Router).handle.func1(0x380b010, 0xc001680f30, 0xc001913100, 0x0, 0x0, 0x0)
    /usr/share/hubspot/build/workspace/InfraOrch/Docker/promxy/gopath/src/github.com/jacksontj/promxy/vendor/github.com/prometheus/common/route/route.go:83 +0x27f
github.com/julienschmidt/httprouter.(*Router).ServeHTTP(0xc000099f20, 0x380b010, 0xc001680f30, 0xc001913100)
    /usr/share/hubspot/build/workspace/InfraOrch/Docker/promxy/gopath/src/github.com/jacksontj/promxy/vendor/github.com/julienschmidt/httprouter/router.go:387 +0xc7e
github.com/prometheus/common/route.(*Router).ServeHTTP(...)
    /usr/share/hubspot/build/workspace/InfraOrch/Docker/promxy/gopath/src/github.com/jacksontj/promxy/vendor/github.com/prometheus/common/route/route.go:121
main.main.func5(0x380b010, 0xc001680f30, 0xc001913100)
    /usr/share/hubspot/build/workspace/InfraOrch/Docker/promxy/gopath/src/github.com/jacksontj/promxy/cmd/promxy/main.go:410 +0x1ce
net/http.HandlerFunc.ServeHTTP(0xc0004b29e0, 0x380b010, 0xc001680f30, 0xc001913100)
    /usr/local/go/src/net/http/server.go:2049 +0x44
github.com/julienschmidt/httprouter.(*Router).ServeHTTP(0xc000c62de0, 0x380b010, 0xc001680f30, 0xc001913100)
    /usr/share/hubspot/build/workspace/InfraOrch/Docker/promxy/gopath/src/github.com/jacksontj/promxy/vendor/github.com/julienschmidt/httprouter/router.go:448 +0x1d2
github.com/jacksontj/promxy/pkg/logging.(*ApacheLoggingHandler).runHandler(0xc000e021b0, 0x380b010, 0xc001680f30, 0xc001913100, 0x0, 0x0)
    /usr/share/hubspot/build/workspace/InfraOrch/Docker/promxy/gopath/src/github.com/jacksontj/promxy/pkg/logging/logging.go:142 +0x85
github.com/jacksontj/promxy/pkg/logging.(*ApacheLoggingHandler).ServeHTTP(0xc000e021b0, 0x38155a0, 0xc001344000, 0xc001913100)
    /usr/share/hubspot/build/workspace/InfraOrch/Docker/promxy/gopath/src/github.com/jacksontj/promxy/pkg/logging/logging.go:165 +0x225
net/http.serverHandler.ServeHTTP(0xc00019a2a0, 0x38155a0, 0xc001344000, 0xc001913100)
    /usr/local/go/src/net/http/server.go:2867 +0xa3
net/http.(*conn).serve(0xc0012bdf40, 0x381e198, 0xc001259cc0)
    /usr/local/go/src/net/http/server.go:1932 +0x8cd
created by net/http.(*Server).Serve
    /usr/local/go/src/net/http/server.go:2993 +0x39b

@jacksontj in our case, we don't actually currently care about exemplar support, but this gives us large error output when we load grafana dashboards due to that panic. So prior to adding support for exemplars, I'm wondering if there's a simple fix for the panic. I'm guessing this only happens when access logs are enabled?

[jacksontj]

@PaulFurtado https://github.com/jacksontj/promxy/pull/475 fixes the panic issue

[sushovan23]

@jacksontj any plans on supporting exemplars? More and more time series DBs are adding support for it. It would great if promxy would also query it

[jacksontj]

@sushovan23 definitely interested in adding support. I don't have a driving need for it in any of my production systems at my day job; so this is falling to my "free time". So it is on the list to get to -- but no promises on when.

The part I expect will take me the most time is understanding the exemplar API and setup the most (as I don't see a public endpoint with exemplar support; so I believe I have to make some local E2E environment to do testing).

So if anyone is interested in contributing a PR, contributing an example environment, or sponsoring the feature -- any/all would be welcome :)

[sushovan23]

@jacksontj I have found this local setup which can be used to have sample app which produced exemplar data and feeding it into prometheus https://vbehar.medium.com/using-prometheus-exemplars-to-jump-from-metrics-to-traces-in-grafana-249e721d4192

example is available here -> https://github.com/prometheus/client_golang/blob/main/examples/random/main.go#L99