search

kubernetes-sigs / prometheus-adapter

An implementation of the custom.metrics.k8s.io API using Prometheus
Apache License 2.0
1.9k stars 551 forks source link

Panic slice out of bands when querying namespace metric #635

Open pmalek opened 7 months ago

pmalek commented 7 months ago

What happened?:

When playing around with config and querying a namespace metric I got a panic:

E0117 21:06:41.565151       1 wrap.go:58] "apiserver panic'd" method="GET" URI="/apis/custom.metrics.k8s.io/v1beta1/namespaces/kong/kong_upstream_latency_ms" auditID="8de84528-caec-4193-b41a-e146d2d871a1"
http2: panic serving 10.244.0.1:38584: runtime error: slice bounds out of range [2:1]
goroutine 1465 [running]:
k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP.func1.1()
        /go/pkg/mod/k8s.io/apiserver@v0.27.2/pkg/server/filters/timeout.go:110 +0x9c
panic({0x1bdd200, 0x4002fd7560})
        /usr/local/go/src/runtime/panic.go:884 +0x1f4
sigs.k8s.io/custom-metrics-apiserver/pkg/apiserver/endpoints/handlers.getRequestOptions(_, {{0x210cbd0, 0x4005a3bab8}, {0x210c660, 0x4005a58380}, {0x20f5f80, 0x4005a58300}, {0x0, 0x0, 0x0}, ...}, ...)
        /go/pkg/mod/sigs.k8s.io/custom-metrics-apiserver@v1.27.0/pkg/apiserver/endpoints/handlers/get.go:146 +0x3ac
sigs.k8s.io/custom-metrics-apiserver/pkg/apiserver/endpoints/handlers.ListResourceWithOptions.func1({0x210c8d0, 0x4005a6b100}, 0x4003566100)
        /go/pkg/mod/sigs.k8s.io/custom-metrics-apiserver@v1.27.0/pkg/apiserver/endpoints/handlers/get.go:109 +0x4a8
sigs.k8s.io/custom-metrics-apiserver/pkg/apiserver/installer.restfulListResourceWithOptions.func1(0x4005a6b080, 0x40001ee930)
        /go/pkg/mod/sigs.k8s.io/custom-metrics-apiserver@v1.27.0/pkg/apiserver/installer/installer.go:280 +0x94
k8s.io/apiserver/pkg/endpoints/metrics.InstrumentRouteFunc.func1(0x4005a6b080, 0x40001ee930)
        /go/pkg/mod/k8s.io/apiserver@v0.27.2/pkg/endpoints/metrics/metrics.go:571 +0x1c8
github.com/emicklei/go-restful/v3.(*Container).dispatch(0x4005a24ab0, {0x210c8d0, 0x4005a6af40}, 0x4003566100)
        /go/pkg/mod/github.com/emicklei/go-restful/v3@v3.10.2/container.go:299 +0x470
github.com/emicklei/go-restful/v3.(*Container).Dispatch(...)
        /go/pkg/mod/github.com/emicklei/go-restful/v3@v3.10.2/container.go:204
k8s.io/apiserver/pkg/server.director.ServeHTTP({{0x1d671da?, 0x116b334?}, 0x4005a24ab0?, 0x4000784b60?}, {0x210c8d0, 0x4005a6af40}, 0x4003566100)
        /go/pkg/mod/k8s.io/apiserver@v0.27.2/pkg/server/handler.go:146 +0x46c
k8s.io/apiserver/pkg/endpoints/filterlatency.trackCompleted.func1({0x210c8d0, 0x4005a6af40}, 0x4003566100)
        /go/pkg/mod/k8s.io/apiserver@v0.27.2/pkg/endpoints/filterlatency/filterlatency.go:110 +0x174
net/http.HandlerFunc.ServeHTTP(0x210d5b8?, {0x210c8d0?, 0x4005a6af40?}, 0x4?)
        /usr/local/go/src/net/http/server.go:2122 +0x38
k8s.io/apiserver/pkg/endpoints/filters.WithAuthorization.func1({0x210c8d0, 0x4005a6af40}, 0x4003566100)
        /go/pkg/mod/k8s.io/apiserver@v0.27.2/pkg/endpoints/filters/authorization.go:64 +0x39c
net/http.HandlerFunc.ServeHTTP(0x433a3342b2?, {0x210c8d0?, 0x4005a6af40?}, 0x912280?)
        /usr/local/go/src/net/http/server.go:2122 +0x38
k8s.io/apiserver/pkg/endpoints/filterlatency.trackStarted.func1({0x210c8d0, 0x4005a6af40}, 0x4003566100)
        /go/pkg/mod/k8s.io/apiserver@v0.27.2/pkg/endpoints/filterlatency/filterlatency.go:84 +0x150
net/http.HandlerFunc.ServeHTTP(0x327c3e0?, {0x210c8d0?, 0x4005a6af40?}, 0x4?)
        /usr/local/go/src/net/http/server.go:2122 +0x38
k8s.io/apiserver/pkg/server/filters.WithMaxInFlightLimit.func1({0x210c8d0, 0x4005a6af40}, 0x4003566100)
        /go/pkg/mod/k8s.io/apiserver@v0.27.2/pkg/server/filters/maxinflight.go:197 +0x22c
net/http.HandlerFunc.ServeHTTP(0x32dde98?, {0x210c8d0?, 0x4005a6af40?}, 0x12c50d4?)
        /usr/local/go/src/net/http/server.go:2122 +0x38
k8s.io/apiserver/pkg/endpoints/filterlatency.trackCompleted.func1({0x210c8d0, 0x4005a6af40}, 0x4003566100)
        /go/pkg/mod/k8s.io/apiserver@v0.27.2/pkg/endpoints/filterlatency/filterlatency.go:110 +0x174
net/http.HandlerFunc.ServeHTTP(0x4007439208?, {0x210c8d0?, 0x4005a6af40?}, 0x163?)
        /usr/local/go/src/net/http/server.go:2122 +0x38
k8s.io/apiserver/pkg/endpoints/filters.WithImpersonation.func1({0x210c8d0, 0x4005a6af40}, 0x4003566100)
        /go/pkg/mod/k8s.io/apiserver@v0.27.2/pkg/endpoints/filters/impersonation.go:50 +0x1a4
net/http.HandlerFunc.ServeHTTP(0x20f2498?, {0x210c8d0?, 0x4005a6af40?}, 0xa?)
        /usr/local/go/src/net/http/server.go:2122 +0x38
k8s.io/apiserver/pkg/endpoints/filterlatency.trackStarted.func1({0x210c8d0, 0x4005a6af40}, 0x4003566100)
        /go/pkg/mod/k8s.io/apiserver@v0.27.2/pkg/endpoints/filterlatency/filterlatency.go:84 +0x150
net/http.HandlerFunc.ServeHTTP(0x32dde98?, {0x210c8d0?, 0x4005a6af40?}, 0x12c50d4?)
        /usr/local/go/src/net/http/server.go:2122 +0x38
k8s.io/apiserver/pkg/endpoints/filterlatency.trackCompleted.func1({0x210c8d0, 0x4005a6af40}, 0x4003566100)
        /go/pkg/mod/k8s.io/apiserver@v0.27.2/pkg/endpoints/filterlatency/filterlatency.go:110 +0x174
net/http.HandlerFunc.ServeHTTP(0x32a7660?, {0x210c8d0?, 0x4005a6af40?}, 0xa?)
        /usr/local/go/src/net/http/server.go:2122 +0x38
k8s.io/apiserver/pkg/endpoints/filterlatency.trackStarted.func1({0x210c8d0, 0x4005a6af40}, 0x4003566100)
        /go/pkg/mod/k8s.io/apiserver@v0.27.2/pkg/endpoints/filterlatency/filterlatency.go:84 +0x150
net/http.HandlerFunc.ServeHTTP(0x32dde98?, {0x210c8d0?, 0x4005a6af40?}, 0x12c50d4?)
        /usr/local/go/src/net/http/server.go:2122 +0x38
k8s.io/apiserver/pkg/endpoints/filterlatency.trackCompleted.func1({0x210c8d0, 0x4005a6af40}, 0x4003566100)
        /go/pkg/mod/k8s.io/apiserver@v0.27.2/pkg/endpoints/filterlatency/filterlatency.go:110 +0x174
net/http.HandlerFunc.ServeHTTP(0x210d5b8?, {0x210c8d0?, 0x4005a6af40?}, 0x20dd120?)
        /usr/local/go/src/net/http/server.go:2122 +0x38
k8s.io/apiserver/pkg/endpoints/filters.withAuthentication.func1({0x210c8d0, 0x4005a6af40}, 0x4003566100)
        /go/pkg/mod/k8s.io/apiserver@v0.27.2/pkg/endpoints/filters/authentication.go:105 +0x504
net/http.HandlerFunc.ServeHTTP(0x210d5b8?, {0x210c8d0?, 0x4005a6af40?}, 0x20e1cc0?)
        /usr/local/go/src/net/http/server.go:2122 +0x38
k8s.io/apiserver/pkg/endpoints/filterlatency.trackStarted.func1({0x210c8d0, 0x4005a6af40}, 0x400354df00)
        /go/pkg/mod/k8s.io/apiserver@v0.27.2/pkg/endpoints/filterlatency/filterlatency.go:94 +0x2f4
net/http.HandlerFunc.ServeHTTP(0x4001378780?, {0x210c8d0?, 0x4005a6af40?}, 0x7cf54?)
        /usr/local/go/src/net/http/server.go:2122 +0x38
k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP.func1()
        /go/pkg/mod/k8s.io/apiserver@v0.27.2/pkg/server/filters/timeout.go:115 +0x68
created by k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP
        /go/pkg/mod/k8s.io/apiserver@v0.27.2/pkg/server/filters/timeout.go:101 +0x188

goroutine 1464 [running]:
golang.org/x/net/http2.(*serverConn).runHandler.func1()
        /go/pkg/mod/golang.org/x/net@v0.8.0/http2/server.go:2304 +0x138
panic({0x1903ca0, 0x4003535810})
        /usr/local/go/src/runtime/panic.go:884 +0x1f4
k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x40038c8cd0, 0x1, 0x400736cc60?})
        /go/pkg/mod/k8s.io/apimachinery@v0.27.2/pkg/util/runtime/runtime.go:56 +0xe0
panic({0x1903ca0, 0x4003535810})
        /usr/local/go/src/runtime/panic.go:884 +0x1f4
k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP(0x40046c8db0, {0x210c8d0, 0x4005a6ae80}, 0xdf8475800?)
        /go/pkg/mod/k8s.io/apiserver@v0.27.2/pkg/server/filters/timeout.go:121 +0x2d4
k8s.io/apiserver/pkg/endpoints/filters.withRequestDeadline.func1({0x210c8d0, 0x4005a6ae80}, 0x400354dd00)
        /go/pkg/mod/k8s.io/apiserver@v0.27.2/pkg/endpoints/filters/request_deadline.go:100 +0x298
net/http.HandlerFunc.ServeHTTP(0xffff7db645d8?, {0x210c8d0?, 0x4005a6ae80?}, 0x1cc4c?)
        /usr/local/go/src/net/http/server.go:2122 +0x38
k8s.io/apiserver/pkg/server/filters.withWaitGroup.func1({0x210c8d0, 0x4005a6ae80}, 0x400354dd00)
        /go/pkg/mod/k8s.io/apiserver@v0.27.2/pkg/server/filters/waitgroup.go:86 +0x18c
net/http.HandlerFunc.ServeHTTP(0x210d5b8?, {0x210c8d0?, 0x4005a6ae80?}, 0x20e1cc0?)
        /usr/local/go/src/net/http/server.go:2122 +0x38
k8s.io/apiserver/pkg/endpoints/filters.WithWarningRecorder.func1({0x210c8d0?, 0x4005a6ae80}, 0x400354dc00)
        /go/pkg/mod/k8s.io/apiserver@v0.27.2/pkg/endpoints/filters/warning.go:35 +0x180
net/http.HandlerFunc.ServeHTTP(0x1b833a0?, {0x210c8d0?, 0x4005a6ae80?}, 0xd?)
        /usr/local/go/src/net/http/server.go:2122 +0x38
k8s.io/apiserver/pkg/endpoints/filters.WithCacheControl.func1({0x210c8d0, 0x4005a6ae80}, 0xffff7df188d0?)
        /go/pkg/mod/k8s.io/apiserver@v0.27.2/pkg/endpoints/filters/cachecontrol.go:31 +0x118
net/http.HandlerFunc.ServeHTTP(0x210e268?, {0x210c8d0?, 0x4005a6ae80?}, 0x20e1cc0?)
        /usr/local/go/src/net/http/server.go:2122 +0x38
k8s.io/apiserver/pkg/server/httplog.withLogging.func1({0x210cff0, 0x4005d16f40}, 0x400354db00)
        /go/pkg/mod/k8s.io/apiserver@v0.27.2/pkg/server/httplog/httplog.go:131 +0x28c
net/http.HandlerFunc.ServeHTTP(0x210d5b8?, {0x210cff0?, 0x4005d16f40?}, 0x20e1cc0?)
        /usr/local/go/src/net/http/server.go:2122 +0x38
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.(*Handler).ServeHTTP(0x40000e48f0, {0x210c8d0?, 0x4005a6ad40}, 0x400354d700)
        /go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.35.1/handler.go:204 +0xe6c
k8s.io/apiserver/pkg/endpoints/filters.WithLatencyTrackers.func1({0x20fed10?, 0x4003a7e028}, 0x400354d600)
        /go/pkg/mod/k8s.io/apiserver@v0.27.2/pkg/endpoints/filters/webhook_duration.go:57 +0x23c
net/http.HandlerFunc.ServeHTTP(0x210d5b8?, {0x20fed10?, 0x4003a7e028?}, 0x20e1cc0?)
        /usr/local/go/src/net/http/server.go:2122 +0x38
k8s.io/apiserver/pkg/endpoints/filters.WithRequestInfo.func1({0x20fed10, 0x4003a7e028}, 0x400354d500)
        /go/pkg/mod/k8s.io/apiserver@v0.27.2/pkg/endpoints/filters/requestinfo.go:39 +0x1b8
net/http.HandlerFunc.ServeHTTP(0x210d5b8?, {0x20fed10?, 0x4003a7e028?}, 0x433a2e934c?)
        /usr/local/go/src/net/http/server.go:2122 +0x38
k8s.io/apiserver/pkg/endpoints/filters.withRequestReceivedTimestampWithClock.func1({0x20fed10, 0x4003a7e028}, 0x400354d400)
        /go/pkg/mod/k8s.io/apiserver@v0.27.2/pkg/endpoints/filters/request_received_time.go:38 +0x13c
net/http.HandlerFunc.ServeHTTP(0x4002c36ab8?, {0x20fed10?, 0x4003a7e028?}, 0xffffa54c0108?)
        /usr/local/go/src/net/http/server.go:2122 +0x38
k8s.io/apiserver/pkg/endpoints/filters.WithMuxAndDiscoveryComplete.func1({0x20fed10, 0x4003a7e028}, 0x400354d400)
        /go/pkg/mod/k8s.io/apiserver@v0.27.2/pkg/endpoints/filters/mux_discovery_complete.go:52 +0x180
net/http.HandlerFunc.ServeHTTP(0xffff7dbcece8?, {0x20fed10?, 0x4003a7e028?}, 0x4000844400?)
        /usr/local/go/src/net/http/server.go:2122 +0x38
k8s.io/apiserver/pkg/server/filters.withPanicRecovery.func1({0x20fed10?, 0x4003a7e028?}, 0x4002f9b101?)
        /go/pkg/mod/k8s.io/apiserver@v0.27.2/pkg/server/filters/wrap.go:74 +0x9c
net/http.HandlerFunc.ServeHTTP(0x1b833a0?, {0x20fed10?, 0x4003a7e028?}, 0x8?)
        /usr/local/go/src/net/http/server.go:2122 +0x38
k8s.io/apiserver/pkg/endpoints/filters.withAuditInit.func1({0x20fed10, 0x4003a7e028}, 0x400354d300)
        /go/pkg/mod/k8s.io/apiserver@v0.27.2/pkg/endpoints/filters/audit_init.go:63 +0x240
net/http.HandlerFunc.ServeHTTP(0x1d4a967?, {0x20fed10?, 0x4003a7e028?}, 0x3?)
        /usr/local/go/src/net/http/server.go:2122 +0x38
k8s.io/apiserver/pkg/server.(*APIServerHandler).ServeHTTP(0x0?, {0x20fed10?, 0x4003a7e028?}, 0x1c7fc?)
        /go/pkg/mod/k8s.io/apiserver@v0.27.2/pkg/server/handler.go:189 +0x30
net/http.serverHandler.ServeHTTP({0x4000058058?}, {0x20fed10, 0x4003a7e028}, 0x400354d300)
        /usr/local/go/src/net/http/server.go:2936 +0x2c0
net/http.initALPNRequest.ServeHTTP({{0x210d5b8?, 0x4007340840?}, 0x4000a14a80?, {0x400618e0f0?}}, {0x20fed10, 0x4003a7e028}, 0x400354d300)
        /usr/local/go/src/net/http/server.go:3545 +0x1d0
golang.org/x/net/http2.(*serverConn).runHandler(0x0?, 0x7cf54?, 0x4007258c00?, 0x0?)
        /go/pkg/mod/golang.org/x/net@v0.8.0/http2/server.go:2311 +0x78
created by golang.org/x/net/http2.(*serverConn).processHeaders
        /go/pkg/mod/golang.org/x/net@v0.8.0/http2/server.go:2025 +0x58c
I0117 21:06:42.704196       1 httplog.go:132] "HTTP" verb="GET" URI="/healthz" latency="143.919µs" userAgent="kube-probe/1.28" audit-ID="7f1ca763-2e93-4173-a77e-9abbb5ff9af2" srcIP="10.244.0.1:41000" resp=200
I0117 21:06:42.704485       1 httplog.go:132] "HTTP" verb="GET" URI="/healthz" latency="67.584µs" userAgent="kube-probe/1.28" audit-ID="84404544-fe98-4b6e-bd6b-49b747cb8af2" srcIP="10.244.0.1:40990" resp=200
I0117 21:06:45.829907       1 httplog.go:132] "HTTP" verb="GET" URI="/apis/custom.metrics.k8s.io/v1beta1/namespaces/kong/services/%2A/kong_upstream_latency_ms" latency="4.124525ms" userAgent="kubectl/v1.29.0 (darwin/arm64) kubernetes/3f7a50f" audit-ID="2bb83135-de14-4236-9b5c-e512d11625d1" srcIP="10.244.0.1:38584" resp=404
I0117 21:06:48.453310       1 httplog.go:132] "HTTP" verb="GET" URI="/apis/custom.metrics.k8s.io/v1beta1/namespaces/kong/pods/%2A/kong_upstream_latency_ms" latency="7.282783ms" userAgent="kubectl/v1.29.0 (darwin/arm64) kubernetes/3f7a50f" audit-ID="68f2e92d-dac0-4a71-b7bf-84be159f4c12" srcIP="10.244.0.1:38584" resp=404

What did you expect to happen?:

No panic.

Please provide the prometheus-adapter config:

prometheus-adapter config ``` prometheus: url: http://prometheus-kube-prometheus-prometheus.kong.svc rules: custom: - seriesQuery: '{__name__=~"^kong_upstream_latency_ms_"}' resources: template: "<<.Resource>>." overrides: namespace: {resource: "namespace"} pod: {resource: "pod"} name: as: "kong_upstream_latency_ms" metricsQuery: |- sum(rate(kong_upstream_latency_ms_sum{route=~".*httproute.*"}[5m])) by (exported_service) / sum(rate(kong_upstream_latency_ms_count{route=~".*httproute.*"}[5m])) by (exported_service) ```

Anything else we need to know?:

Environment:

dgrisonnet commented 7 months ago

/triage accepted /assign