search

grafana / metrictank

metrics2.0 based, multi-tenant timeseries store for Graphite and friends.
GNU Affero General Public License v3.0
622 stars 104 forks source link

Panic in Graphite Proxy #1860

Closed replay closed 3 years ago

replay commented 4 years ago

I just observed a panic in the graphite proxy, I don't know yet what the exact circumstances were which have caused it

Metrictank Version: v0.13.1-813-g4610de9 Golang Version: go1.11.4 OS: GKE node version v1.14.10-gke.17

Here is the stack trace:

  |   | 2020-07-23 16:36:28 | /usr/local/go/src/net/http/server.go:2851 +0x2f5
  |   | 2020-07-23 16:36:28 | created by net/http.(*Server).Serve
  |   | 2020-07-23 16:36:28 | /usr/local/go/src/net/http/server.go:1847 +0x646
  |   | 2020-07-23 16:36:28 | net/http.(*conn).serve(0xc000190500, 0x118b2c0, 0xc0000420c0)
  |   | 2020-07-23 16:36:28 | /usr/local/go/src/net/http/server.go:2741 +0xab
  |   | 2020-07-23 16:36:28 | net/http.serverHandler.ServeHTTP(0xc0000a69c0, 0x118a100, 0xc0002e02a0, 0xc0001b0700)
  |   | 2020-07-23 16:36:28 | /go/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/macaron.go:220 +0x132
  |   | 2020-07-23 16:36:28 | github.com/grafana/metrictank/vendor/gopkg.in/macaron%2ev1.(*Macaron).ServeHTTP(0xc00014e680, 0x118a100, 0xc0002e02a0, 0xc0001b0700)
  |   | 2020-07-23 16:36:28 | /go/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/router.go:294 +0x257
  |   | 2020-07-23 16:36:28 | github.com/grafana/metrictank/vendor/gopkg.in/macaron%2ev1.(*Router).ServeHTTP(0xc000084f60, 0x118a100, 0xc0002e02a0, 0xc0001b0700)
  |   | 2020-07-23 16:36:28 | /go/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/router.go:187 +0x298
  |   | 2020-07-23 16:36:28 | github.com/grafana/metrictank/vendor/gopkg.in/macaron%2ev1.(*Router).Handle.func1(0x118a100, 0xc0002e02a0, 0xc0001b0700, 0x0)
  |   | 2020-07-23 16:36:28 | /go/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/context.go:121 +0x81
  |   | 2020-07-23 16:36:28 | github.com/grafana/metrictank/vendor/gopkg.in/macaron%2ev1.(*Context).run(0xc000904000)
  |   | 2020-07-23 16:36:28 | /go/src/github.com/grafana/metrictank/vendor/github.com/go-macaron/inject/inject.go:135 +0x1bc
  |   | 2020-07-23 16:36:28 | github.com/grafana/metrictank/vendor/github.com/go-macaron/inject.(*injector).Invoke(0xc000640440, 0xeddb40, 0x1087a08, 0xc00037e370, 0xc000172690, 0x50, 0xc000030a80, 0xc000f89bc8)
  |   | 2020-07-23 16:36:28 | /go/src/github.com/grafana/metrictank/vendor/github.com/go-macaron/inject/inject.go:157 +0x231
  |   | 2020-07-23 16:36:28 | github.com/grafana/metrictank/vendor/github.com/go-macaron/inject.(*injector).fastInvoke(0xc000640440, 0x7fe614054818, 0x1087a08, 0x119d8a0, 0xeddb40, 0x2, 0xc000f89b98, 0x40bdaf, 0xc00037e2d0, 0xf0, ...)
  |   | 2020-07-23 16:36:28 | /go/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/logger.go:40 +0x74
  |   | 2020-07-23 16:36:28 | github.com/grafana/metrictank/vendor/gopkg.in/macaron%2ev1.LoggerInvoker.Invoke(0x1087a08, 0xc000640460, 0x2, 0x2, 0xc000172320, 0x16, 0x7fe614054818, 0xc00061a018, 0x940000c000f89b30)
  |   | 2020-07-23 16:36:28 | /go/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/recovery.go:161 +0x5b
  |   | 2020-07-23 16:36:28 | github.com/grafana/metrictank/vendor/gopkg.in/macaron%2ev1.Recovery.func1(0xc000904000, 0xc000172320)
  |   | 2020-07-23 16:36:28 | /go/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/context.go:112 +0x2f
  |   | 2020-07-23 16:36:28 | github.com/grafana/metrictank/vendor/gopkg.in/macaron%2ev1.(*Context).Next(0xc000904000)
  |   | 2020-07-23 16:36:28 | /go/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/context.go:121 +0x81
  |   | 2020-07-23 16:36:28 | github.com/grafana/metrictank/vendor/gopkg.in/macaron%2ev1.(*Context).run(0xc000904000)
  |   | 2020-07-23 16:36:28 | /go/src/github.com/grafana/metrictank/vendor/github.com/go-macaron/inject/inject.go:135 +0x1bc
  |   | 2020-07-23 16:36:28 | github.com/grafana/metrictank/vendor/github.com/go-macaron/inject.(*injector).Invoke(0xc000640440, 0xec76c0, 0xc0001606b0, 0x0, 0x0, 0x0, 0x0, 0x0)
  |   | 2020-07-23 16:36:28 | /go/src/github.com/grafana/metrictank/vendor/github.com/go-macaron/inject/inject.go:157 +0x231
  |   | 2020-07-23 16:36:28 | github.com/grafana/metrictank/vendor/github.com/go-macaron/inject.(*injector).fastInvoke(0xc000640440, 0x7fe614054838, 0xc0001606b0, 0x119d8a0, 0xec76c0, 0x1, 0x0, 0x0, 0x0, 0x0, ...)
  |   | 2020-07-23 16:36:28 | /go/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/context.go:79 +0x51
  |   | 2020-07-23 16:36:28 | github.com/grafana/metrictank/vendor/gopkg.in/macaron%2ev1.ContextInvoker.Invoke(0xc0001606b0, 0xc000161ce0, 0x1, 0x1, 0xc000904000, 0x16, 0x7fe614054838, 0x0, 0x0)
  |   | 2020-07-23 16:36:28 | /go/src/github.com/grafana/metrictank/api/middleware/stats.go:76 +0xa9
  |   | 2020-07-23 16:36:28 | github.com/grafana/metrictank/api/middleware.RequestStats.func1(0xc000904000)
  |   | 2020-07-23 16:36:28 | /go/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/context.go:112 +0x2f
  |   | 2020-07-23 16:36:28 | github.com/grafana/metrictank/vendor/gopkg.in/macaron%2ev1.(*Context).Next(0xc000904000)
  |   | 2020-07-23 16:36:28 | /go/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/context.go:121 +0x81
  |   | 2020-07-23 16:36:28 | github.com/grafana/metrictank/vendor/gopkg.in/macaron%2ev1.(*Context).run(0xc000904000)
  |   | 2020-07-23 16:36:28 | /go/src/github.com/grafana/metrictank/vendor/github.com/go-macaron/inject/inject.go:135 +0x1bc
  |   | 2020-07-23 16:36:28 | github.com/grafana/metrictank/vendor/github.com/go-macaron/inject.(*injector).Invoke(0xc000640440, 0xec76c0, 0xc0001503e0, 0x40d7e6, 0xc000f89798, 0xc00061a008, 0x44bd38, 0x8e13026b35acf)
  |   | 2020-07-23 16:36:28 | /go/src/github.com/grafana/metrictank/vendor/github.com/go-macaron/inject/inject.go:157 +0x231
  |   | 2020-07-23 16:36:28 | github.com/grafana/metrictank/vendor/github.com/go-macaron/inject.(*injector).fastInvoke(0xc000640440, 0x7fe614054838, 0xc0001503e0, 0x119d8a0, 0xec76c0, 0x1, 0xc000f896c8, 0xc000f896d0, 0x1, 0x8, ...)
  |   | 2020-07-23 16:36:28 | /go/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/context.go:79 +0x51
  |   | 2020-07-23 16:36:28 | github.com/grafana/metrictank/vendor/gopkg.in/macaron%2ev1.ContextInvoker.Invoke(0xc0001503e0, 0xc000161cf0, 0x1, 0x1, 0xc000904000, 0x16, 0x7fe614054838, 0xa, 0x17)
  |   | 2020-07-23 16:36:28 | /go/src/github.com/grafana/metrictank/api/middleware/tracer.go:70 +0x6a6
  |   | 2020-07-23 16:36:28 | github.com/grafana/metrictank/api/middleware.Tracer.func1(0xc000904000)
  |   | 2020-07-23 16:36:28 | /go/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/context.go:112 +0x2f
  |   | 2020-07-23 16:36:28 | github.com/grafana/metrictank/vendor/gopkg.in/macaron%2ev1.(*Context).Next(0xc000904000)
  |   | 2020-07-23 16:36:28 | /go/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/context.go:121 +0x81
  |   | 2020-07-23 16:36:28 | github.com/grafana/metrictank/vendor/gopkg.in/macaron%2ev1.(*Context).run(0xc000904000)
  |   | 2020-07-23 16:36:28 | /go/src/github.com/grafana/metrictank/vendor/github.com/go-macaron/inject/inject.go:137 +0xca
  |   | 2020-07-23 16:36:28 | github.com/grafana/metrictank/vendor/github.com/go-macaron/inject.(*injector).Invoke(0xc000640440, 0xe5b960, 0x1085b60, 0x0, 0x0, 0x0, 0x0, 0x0)
  |   | 2020-07-23 16:36:28 | /go/src/github.com/grafana/metrictank/vendor/github.com/go-macaron/inject/inject.go:177 +0x2a4
  |   | 2020-07-23 16:36:28 | github.com/grafana/metrictank/vendor/github.com/go-macaron/inject.(*injector).callInvoke(0xc000640440, 0xe5b960, 0x1085b60, 0x119d8a0, 0xe5b960, 0x1, 0x0, 0x0, 0x0, 0x0, ...)
  |   | 2020-07-23 16:36:28 | /usr/local/go/src/reflect/value.go:308 +0xa4
  |   | 2020-07-23 16:36:28 | reflect.Value.Call(0xe5b960, 0x1085b60, 0x13, 0xc0006404c0, 0x1, 0x1, 0x0, 0x0, 0x0)
  |   | 2020-07-23 16:36:28 | /usr/local/go/src/reflect/value.go:447 +0x454
  |   | 2020-07-23 16:36:28 | reflect.Value.call(0xe5b960, 0x1085b60, 0x13, 0x1034980, 0x4, 0xc0006404c0, 0x1, 0x1, 0x102fe00, 0xa8f001, ...)
  |   | 2020-07-23 16:36:28 | /go/src/github.com/grafana/metrictank/api/middleware/logger.go:45 +0x150
  |   | 2020-07-23 16:36:28 | github.com/grafana/metrictank/api/middleware.Logger.func1(0xc0006404a0)
  |   | 2020-07-23 16:36:28 | /go/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/context.go:112 +0x2f
  |   | 2020-07-23 16:36:28 | github.com/grafana/metrictank/vendor/gopkg.in/macaron%2ev1.(*Context).Next(0xc000904000)
  |   | 2020-07-23 16:36:28 | /go/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/context.go:121 +0x81
  |   | 2020-07-23 16:36:28 | github.com/grafana/metrictank/vendor/gopkg.in/macaron%2ev1.(*Context).run(0xc000904000)
  |   | 2020-07-23 16:36:28 | /go/src/github.com/grafana/metrictank/vendor/github.com/go-macaron/inject/inject.go:135 +0x1bc
  |   | 2020-07-23 16:36:28 | github.com/grafana/metrictank/vendor/github.com/go-macaron/inject.(*injector).Invoke(0xc000640440, 0xeddb40, 0x1087a08, 0xc000f88b68, 0xa73d7f, 0xea4e20, 0xc0009f0690, 0xc000f88b58)
  |   | 2020-07-23 16:36:28 | /go/src/github.com/grafana/metrictank/vendor/github.com/go-macaron/inject/inject.go:157 +0x231
  |   | 2020-07-23 16:36:28 | github.com/grafana/metrictank/vendor/github.com/go-macaron/inject.(*injector).fastInvoke(0xc000640440, 0x7fe614054818, 0x1087a08, 0x119d8a0, 0xeddb40, 0x2, 0x40df3a, 0xfe5be0, 0xc00061a018, 0xc000f88b58, ...)
  |   | 2020-07-23 16:36:28 | /go/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/logger.go:40 +0x74
  |   | 2020-07-23 16:36:28 | github.com/grafana/metrictank/vendor/gopkg.in/macaron%2ev1.LoggerInvoker.Invoke(0x1087a08, 0xc000640500, 0x2, 0x2, 0xc000172320, 0x16, 0x7fe614054818, 0x0, 0x4036ac)
  |   | 2020-07-23 16:36:28 | /go/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/recovery.go:161 +0x5b
  |   | 2020-07-23 16:36:28 | github.com/grafana/metrictank/vendor/gopkg.in/macaron%2ev1.Recovery.func1(0xc000904000, 0xc000172320)
  |   | 2020-07-23 16:36:28 | /go/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/context.go:112 +0x2f
  |   | 2020-07-23 16:36:28 | github.com/grafana/metrictank/vendor/gopkg.in/macaron%2ev1.(*Context).Next(0xc000904000)
  |   | 2020-07-23 16:36:28 | /go/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/context.go:121 +0x81
  |   | 2020-07-23 16:36:28 | github.com/grafana/metrictank/vendor/gopkg.in/macaron%2ev1.(*Context).run(0xc000904000)
  |   | 2020-07-23 16:36:28 | /go/src/github.com/grafana/metrictank/vendor/github.com/go-macaron/inject/inject.go:135 +0x1bc
  |   | 2020-07-23 16:36:28 | github.com/grafana/metrictank/vendor/github.com/go-macaron/inject.(*injector).Invoke(0xc000640440, 0xec76c0, 0x1087448, 0xfe03e0, 0xfe03e0, 0xc000172320, 0x16, 0xfe03e0)
  |   | 2020-07-23 16:36:28 | /go/src/github.com/grafana/metrictank/vendor/github.com/go-macaron/inject/inject.go:157 +0x231
  |   | 2020-07-23 16:36:28 | github.com/grafana/metrictank/vendor/github.com/go-macaron/inject.(*injector).fastInvoke(0xc000640440, 0x7fe614054838, 0x1087448, 0x119d8a0, 0xec76c0, 0x1, 0x1031ea0, 0x1031ea0, 0x1, 0xc000f888c8, ...)
  |   | 2020-07-23 16:36:28 | /go/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/context.go:79 +0x51
  |   | 2020-07-23 16:36:28 | github.com/grafana/metrictank/vendor/gopkg.in/macaron%2ev1.ContextInvoker.Invoke(0x1087448, 0xc000161dc0, 0x1, 0x1, 0xc000904000, 0x16, 0x7fe614054838, 0x0, 0x0)
  |   | 2020-07-23 16:36:28 | /go/src/github.com/grafana/metrictank/vendor/github.com/raintank/gziper/gzip.go:100 +0x16c
  |   | 2020-07-23 16:36:28 | github.com/grafana/metrictank/vendor/github.com/raintank/gziper.Gziper.func1(0xc000904000)
  |   | 2020-07-23 16:36:28 | /go/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/context.go:112 +0x2f
  |   | 2020-07-23 16:36:28 | github.com/grafana/metrictank/vendor/gopkg.in/macaron%2ev1.(*Context).Next(0xc000904000)
  |   | 2020-07-23 16:36:28 | /go/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/context.go:121 +0x81
  |   | 2020-07-23 16:36:28 | github.com/grafana/metrictank/vendor/gopkg.in/macaron%2ev1.(*Context).run(0xc000904000)
  |   | 2020-07-23 16:36:28 | /go/src/github.com/grafana/metrictank/vendor/github.com/go-macaron/inject/inject.go:137 +0xca
  |   | 2020-07-23 16:36:28 | github.com/grafana/metrictank/vendor/github.com/go-macaron/inject.(*injector).Invoke(0xc000640440, 0xe787e0, 0xc000204660, 0x0, 0x0, 0x0, 0x0, 0x0)
  |   | 2020-07-23 16:36:28 | /go/src/github.com/grafana/metrictank/vendor/github.com/go-macaron/inject/inject.go:177 +0x2a4
  |   | 2020-07-23 16:36:28 | github.com/grafana/metrictank/vendor/github.com/go-macaron/inject.(*injector).callInvoke(0xc000640440, 0xe787e0, 0xc000204660, 0x119d8a0, 0xe787e0, 0x2, 0xaf777c, 0xc000640480, 0x0, 0xc00113c668, ...)
  |   | 2020-07-23 16:36:28 | /usr/local/go/src/reflect/value.go:308 +0xa4
  |   | 2020-07-23 16:36:28 | reflect.Value.Call(0xe787e0, 0xc000204660, 0x13, 0xc0009f08a0, 0x2, 0x2, 0x0, 0x0, 0x0)
  |   | 2020-07-23 16:36:28 | /usr/local/go/src/reflect/value.go:447 +0x454
  |   | 2020-07-23 16:36:28 | reflect.Value.call(0xe787e0, 0xc000204660, 0x13, 0x1034980, 0x4, 0xc0009f08a0, 0x2, 0x2, 0x102fe00, 0xa8f001, ...)
  |   | 2020-07-23 16:36:28 | /go/src/github.com/grafana/metrictank/api/routes.go:65 +0x65
  |   | 2020-07-23 16:36:28 | github.com/grafana/metrictank/api.(*Server).renderMetrics-fm(0xc0006404a0, 0xc00014855f, 0x3, 0xc000148569, 0x3, 0x0, 0x0, 0x0, 0x0, 0x18b, ...)
  |   | 2020-07-23 16:36:28 | /go/src/github.com/grafana/metrictank/api/graphite.go:255 +0x1375
  |   | 2020-07-23 16:36:28 | github.com/grafana/metrictank/api.(*Server).renderMetrics(0xc0003ae180, 0xc0006404a0, 0xc00014855f, 0x3, 0xc000148569, 0x3, 0x0, 0x0, 0x0, 0x0, ...)
  |   | 2020-07-23 16:36:28 | /go/src/github.com/grafana/metrictank/api/graphite.go:153 +0x328
  |   | 2020-07-23 16:36:28 | github.com/grafana/metrictank/api.(*Server).proxyToGraphite(0xc0003ae180, 0xc0006404a0)
  |   | 2020-07-23 16:36:28 | /usr/local/go/src/net/http/httputil/reverseproxy.go:284 +0x1058
  |   | 2020-07-23 16:36:28 | net/http/httputil.(*ReverseProxy).ServeHTTP(0xc000172550, 0x1187ec0, 0xc0009f0720, 0xc0001b0a00)
  |   | 2020-07-23 16:36:28 | /usr/local/go/src/runtime/panic.go:513 +0x1b9
  |   | 2020-07-23 16:36:28 | panic(0xeae500, 0xc00007ab20)
  |   | 2020-07-23 16:36:28 | /go/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/recovery.go:156 +0x29e
  |   | 2020-07-23 16:36:28 | github.com/grafana/metrictank/vendor/gopkg.in/macaron%2ev1.Recovery.func1.1(0xc000172320, 0xc000904000)
  |   | 2020-07-23 16:36:28 | /go/src/github.com/grafana/metrictank/vendor/github.com/raintank/gziper/gzip.go:155 +0x4e
  |   | 2020-07-23 16:36:28 | github.com/grafana/metrictank/vendor/github.com/raintank/gziper.(*gzipResponseWriter).Write(0xc0009f0720, 0xc0002ca000, 0x14ff, 0x1500, 0x14ff, 0x1500, 0x14ff)
  |   | 2020-07-23 16:36:28 | /usr/local/go/src/runtime/panic.go:513 +0x1b9
  |   | 2020-07-23 16:36:28 | panic(0xec7c00, 0x1993aa0)
  |   | 2020-07-23 16:36:28 | /go/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/recovery.go:156 +0x29e
  |   | 2020-07-23 16:36:28 | github.com/grafana/metrictank/vendor/gopkg.in/macaron%2ev1.Recovery.func1.1(0xc000172320, 0xc000904000)
  |   | 2020-07-23 16:36:28 | /go/src/github.com/grafana/metrictank/vendor/github.com/raintank/gziper/gzip.go:155 +0x4e
  |   | 2020-07-23 16:36:28 | github.com/grafana/metrictank/vendor/github.com/raintank/gziper.(*gzipResponseWriter).Write(0xc0009f0720, 0xc002813000, 0x16dc, 0x1800, 0x16dc, 0x1800, 0x16dc)
  |   | 2020-07-23 16:36:28 | /usr/local/go/src/runtime/panic.go:513 +0x1b9
  |   | 2020-07-23 16:36:28 | panic(0xec7c00, 0x1993aa0)
  |   | 2020-07-23 16:36:28 | /usr/local/go/src/net/http/server.go:1746 +0xd0
  |   | 2020-07-23 16:36:28 | net/http.(*conn).serve.func1(0xc000190500)
  |   | 2020-07-23 16:36:28 | goroutine 6418193 [running]:
  |   | 2020-07-23 16:36:28 | 2020/07/23 16:36:28 http: panic serving 10.52.68.33:60876: runtime error: invalid memory address or nil pointer dereference
  |   | 2020-07-23 16:36:28 | 2020/07/23 16:36:28 http: multiple response.WriteHeader calls
stale[bot] commented 3 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.