search

thanos-io / thanos

Highly available Prometheus setup with long term storage capabilities. A CNCF Incubating project.
https://thanos.io
Apache License 2.0
13.09k stars 2.1k forks source link

NPE in query app #286

Closed alvaroaleman closed 6 years ago

alvaroaleman commented 6 years ago

This gets triggered when using this grafana dashboard, it must have a query that causes the NPE. If I use prometheus directly as datasource in the grafana, everything is fine.

The stacktrace looks like this:

2018/04/12 18:41:51 http: panic serving 10.244.2.8:44358: runtime error: invalid memory address or nil pointer dereference
goroutine 69307 [running]:
net/http.(*conn).serve.func1(0xc420266000)
    /usr/local/go/src/net/http/server.go:1726 +0xd0
panic(0xd179c0, 0x142cc60)
    /usr/local/go/src/runtime/panic.go:502 +0x229
github.com/improbable-eng/thanos/pkg/query.(*querier).Select(0xc42019e7e0, 0x0, 0xc42000c050, 0x1, 0x1, 0x0, 0x0, 0x0, 0x0)
    /go/src/github.com/improbable-eng/thanos/pkg/query/querier.go:166 +0x134
github.com/improbable-eng/thanos/pkg/query/api.(*API).series(0xc4200783c0, 0xc420290c00, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
    /go/src/github.com/improbable-eng/thanos/pkg/query/api/v1.go:446 +0x8dd
github.com/improbable-eng/thanos/pkg/query/api.(*API).(github.com/improbable-eng/thanos/pkg/query/api.series)-fm(0xc420290c00, 0xc420758480, 0x0, 0xc42008aa98, 0xc42008aa90, 0x11, 0xc420399858)
    /go/src/github.com/improbable-eng/thanos/pkg/query/api/v1.go:165 +0x34
github.com/improbable-eng/thanos/pkg/query/api.(*API).Register.func1.1(0xf08560, 0xc420758480, 0xc420290c00)
    /go/src/github.com/improbable-eng/thanos/pkg/query/api/v1.go:147 +0x55
net/http.HandlerFunc.ServeHTTP(0xc42022e510, 0xf08560, 0xc420758480, 0xc420290c00)
    /usr/local/go/src/net/http/server.go:1947 +0x44
github.com/improbable-eng/thanos/vendor/github.com/NYTimes/gziphandler.GzipHandlerWithOpts.func1.1(0xf096e0, 0xc42000c048, 0xc420290c00)
    /go/src/github.com/improbable-eng/thanos/vendor/github.com/NYTimes/gziphandler/gzip.go:277 +0x179
net/http.HandlerFunc.ServeHTTP(0xc4205d94a0, 0xf096e0, 0xc42000c048, 0xc420290c00)
    /usr/local/go/src/net/http/server.go:1947 +0x44
github.com/improbable-eng/thanos/pkg/tracing.HTTPMiddleware.func1(0xf096e0, 0xc42000c048, 0xc420290b00)
    /go/src/github.com/improbable-eng/thanos/pkg/tracing/http.go:37 +0x577
net/http.HandlerFunc.ServeHTTP(0xc4202394f0, 0xf096e0, 0xc42000c048, 0xc420290b00)
    /usr/local/go/src/net/http/server.go:1947 +0x44
net/http.(Handler).ServeHTTP-fm(0xf096e0, 0xc42000c048, 0xc420290b00)
    /go/src/github.com/improbable-eng/thanos/vendor/github.com/prometheus/client_golang/prometheus/http.go:179 +0x4d
github.com/improbable-eng/thanos/vendor/github.com/prometheus/client_golang/prometheus.InstrumentHandlerFuncWithOpts.func1(0xf0aba0, 0xc420326000, 0xc420290b00)
    /go/src/github.com/improbable-eng/thanos/vendor/github.com/prometheus/client_golang/prometheus/http.go:287 +0x26f
github.com/improbable-eng/thanos/vendor/github.com/prometheus/common/route.(*Router).handle.func1(0xf0aba0, 0xc420326000, 0xc420290a00, 0x0, 0x0, 0x0)
    /go/src/github.com/improbable-eng/thanos/vendor/github.com/prometheus/common/route/route.go:50 +0x222
github.com/improbable-eng/thanos/vendor/github.com/julienschmidt/httprouter.(*Router).ServeHTTP(0xc4202a7230, 0xf0aba0, 0xc420326000, 0xc420290a00)
    /go/src/github.com/improbable-eng/thanos/vendor/github.com/julienschmidt/httprouter/router.go:299 +0x6c1
github.com/improbable-eng/thanos/vendor/github.com/prometheus/common/route.(*Router).ServeHTTP(0xc420104d80, 0xf0aba0, 0xc420326000, 0xc420290a00)
    /go/src/github.com/improbable-eng/thanos/vendor/github.com/prometheus/common/route/route.go:88 +0x4c
net/http.(*ServeMux).ServeHTTP(0xc4205d9bc0, 0xf0aba0, 0xc420326000, 0xc420290a00)
    /usr/local/go/src/net/http/server.go:2337 +0x130
net/http.serverHandler.ServeHTTP(0xc4201f1ba0, 0xf0aba0, 0xc420326000, 0xc420290a00)
    /usr/local/go/src/net/http/server.go:2694 +0xbc
net/http.(*conn).serve(0xc420266000, 0xf0bce0, 0xc42058a280)
    /usr/local/go/src/net/http/server.go:1830 +0x651
created by net/http.(*Server).Serve
    /usr/local/go/src/net/http/server.go:2795 +0x27b

If there is anything more I can do to assist debugging this, just ping me.

alvaroaleman commented 6 years ago

Oh I forgot version info:

$ kubectl exec -it  thanos-query-74fb7fcc86-b4z6z -- thanos --version
thanos, version v0.0.1 (branch: master, revision: 1516d01e7564c901e8f229a57ee40d76b8ceebb1)
  build user:       circleci@28388de946b9
  build date:       20180410-15:01:13
  go version:       go1.10.1
bwplotka commented 6 years ago

Thanks!