panic in promql.(*timeseries).CopyFromShallowTimestamps

[vrischmann]

Describe the bug

I'm using a Histogram and trying to graph it with Grafana using this query:

histogram_quantile($quantile, sum(irate(sender_dial_latencies_bucket{app = "myapp"}[5m])) by (vmrange))

The query works fine for all other histogram I have but for this particular one the HTTP handler goroutine panics.

Here is the panic I get in the logs:

2020-01-27T12:37:23.906+0000        error        net/http/server.go:3053        http: panic serving 10.66.66.66:6878: runtime error: invalid memory address or nil pointer dereference
goroutine 3669807375 [running]:
net/http.(*conn).serve.func1(0xc126962000)
        net/http/server.go:1767 +0x139
panic(0xa78660, 0xf41d20)
        runtime/panic.go:679 +0x1b2
github.com/VictoriaMetrics/VictoriaMetrics/app/vmselect/promql.(*timeseries).CopyFromShallowTimestamps(0xc112d49500, 0x0)
        github.com/VictoriaMetrics/VictoriaMetrics@/app/vmselect/promql/timeseries.go:44 +0x67
github.com/VictoriaMetrics/VictoriaMetrics/app/vmselect/promql.vmrangeBucketsToLE.func1(0x0, 0xb0be9a, 0x4, 0xc18496c700)
        github.com/VictoriaMetrics/VictoriaMetrics@/app/vmselect/promql/transform.go:337 +0x51
github.com/VictoriaMetrics/VictoriaMetrics/app/vmselect/promql.vmrangeBucketsToLE(0xc112d48540, 0xe, 0xe, 0x0, 0x0, 0x0)
        github.com/VictoriaMetrics/VictoriaMetrics@/app/vmselect/promql/transform.go:379 +0xcc4
github.com/VictoriaMetrics/VictoriaMetrics/app/vmselect/promql.transformHistogramQuantile(0xc13e73ec60, 0xc13e73ec60, 0xc18b1f4580, 0x12, 0xc0001abcb8, 0x2)
        github.com/VictoriaMetrics/VictoriaMetrics@/app/vmselect/promql/transform.go:412 +0x11a
github.com/VictoriaMetrics/VictoriaMetrics/app/vmselect/promql.evalExpr(0xc196596540, 0xbd27e0, 0xc1601c98f0, 0xc1601c98f0, 0x0, 0x0, 0x70, 0xc196596540)
        github.com/VictoriaMetrics/VictoriaMetrics@/app/vmselect/promql/eval.go:181 +0x1303
github.com/VictoriaMetrics/VictoriaMetrics/app/vmselect/promql.Exec(0xc196596540, 0xc1581d2380, 0x7b, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
        github.com/VictoriaMetrics/VictoriaMetrics@/app/vmselect/promql/exec.go:58 +0xdf
github.com/VictoriaMetrics/VictoriaMetrics/app/vmselect/prometheus.QueryRangeHandler(0xbdc720, 0xc0b0008ae0, 0xc0ea7e1a00, 0x1, 0x0)
        github.com/VictoriaMetrics/VictoriaMetrics@/app/vmselect/prometheus/prometheus.go:587 +0x3ec
github.com/VictoriaMetrics/VictoriaMetrics/app/vmselect.RequestHandler(0xbdc720, 0xc0b0008ae0, 0xc0ea7e1a00, 0xc060519b00)
        github.com/VictoriaMetrics/VictoriaMetrics@/app/vmselect/main.go:112 +0xcbc
main.requestHandler(0xbdc720, 0xc0b0008ae0, 0xc0ea7e1a00, 0xc0b0008a00)
        github.com/VictoriaMetrics/VictoriaMetrics@/app/victoria-metrics/main.go:56 +0x6c
github.com/VictoriaMetrics/VictoriaMetrics/lib/httpserver.handlerWrapper(0xbdc720, 0xc0b0008ae0, 0xc0ea7e1a00, 0xb3db78)
        github.com/VictoriaMetrics/VictoriaMetrics@/lib/httpserver/httpserver.go:185 +0x1c3
github.com/VictoriaMetrics/VictoriaMetrics/lib/httpserver.gzipHandler.func1(0xbdc720, 0xc0b0008ae0, 0xc0ea7e1a00)
        github.com/VictoriaMetrics/VictoriaMetrics@/lib/httpserver/httpserver.go:144 +0x88
net/http.HandlerFunc.ServeHTTP(0xc092a38680, 0xbdcc20, 0xc082e0fa40, 0xc0ea7e1a00)
        net/http/server.go:2007 +0x44
net/http.serverHandler.ServeHTTP(0xc05cada540, 0xbdcc20, 0xc082e0fa40, 0xc0ea7e1a00)
        net/http/server.go:2802 +0xa4
net/http.(*conn).serve(0xc126962000, 0xbde720, 0xc13f68ac80)
        net/http/server.go:1890 +0x875
created by net/http.(*Server).Serve
        net/http/server.go:2927 +0x38e

To Reproduce

It was enough to hit the grafana URL to query VM:

https://mygrafana/api/datasources/proxy/1/api/v1/query_range?query=histogram_quantile(0.5%2C%20sum(irate(sender_dial_latencies_bucket{app%20%3D%20%22myapp%22}[5m]))%20by%20(vmrange))&start=1580122890&end=1580130090&step=30

However now I'm unable to reproduce the crash. I see in the logs that VM merged some rows, maybe it fixed the problem.

Expected behavior

I expect VM to not panic.

Version

victoria-metrics-20191203-234719-tags-v1.30.4-0-g359c4d61

[valyala]

@vrischmann , thanks for the bug report! The bug has been fixed in the commit 2392a359e1dd5a4cd2215515aa60772b795e8e29 . This commit will be included in the next VictoriaMetrics release. In the mean time you can verify the bugfix by building VictoriaMetrics from sources at the commit 2392a359e1dd5a4cd2215515aa60772b795e8e29 according to these instructions.

[valyala]

@vrischmann , could you verify whether the issue is fixed in v1.32.8?

Feel free re-opening the issue if it isn't fixed yet.

[vrischmann]

Hi, just realized I didn't answer here.

I can confirm the fix works. Thank you !

[valyala]

Thanks for the update!