Open pmalek opened 7 months ago
What happened?:
When playing around with config and querying a namespace metric I got a panic:
E0117 21:06:41.565151 1 wrap.go:58] "apiserver panic'd" method="GET" URI="/apis/custom.metrics.k8s.io/v1beta1/namespaces/kong/kong_upstream_latency_ms" auditID="8de84528-caec-4193-b41a-e146d2d871a1" http2: panic serving 10.244.0.1:38584: runtime error: slice bounds out of range [2:1] goroutine 1465 [running]: k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP.func1.1() /go/pkg/mod/k8s.io/apiserver@v0.27.2/pkg/server/filters/timeout.go:110 +0x9c panic({0x1bdd200, 0x4002fd7560}) /usr/local/go/src/runtime/panic.go:884 +0x1f4 sigs.k8s.io/custom-metrics-apiserver/pkg/apiserver/endpoints/handlers.getRequestOptions(_, {{0x210cbd0, 0x4005a3bab8}, {0x210c660, 0x4005a58380}, {0x20f5f80, 0x4005a58300}, {0x0, 0x0, 0x0}, ...}, ...) /go/pkg/mod/sigs.k8s.io/custom-metrics-apiserver@v1.27.0/pkg/apiserver/endpoints/handlers/get.go:146 +0x3ac sigs.k8s.io/custom-metrics-apiserver/pkg/apiserver/endpoints/handlers.ListResourceWithOptions.func1({0x210c8d0, 0x4005a6b100}, 0x4003566100) /go/pkg/mod/sigs.k8s.io/custom-metrics-apiserver@v1.27.0/pkg/apiserver/endpoints/handlers/get.go:109 +0x4a8 sigs.k8s.io/custom-metrics-apiserver/pkg/apiserver/installer.restfulListResourceWithOptions.func1(0x4005a6b080, 0x40001ee930) /go/pkg/mod/sigs.k8s.io/custom-metrics-apiserver@v1.27.0/pkg/apiserver/installer/installer.go:280 +0x94 k8s.io/apiserver/pkg/endpoints/metrics.InstrumentRouteFunc.func1(0x4005a6b080, 0x40001ee930) /go/pkg/mod/k8s.io/apiserver@v0.27.2/pkg/endpoints/metrics/metrics.go:571 +0x1c8 github.com/emicklei/go-restful/v3.(*Container).dispatch(0x4005a24ab0, {0x210c8d0, 0x4005a6af40}, 0x4003566100) /go/pkg/mod/github.com/emicklei/go-restful/v3@v3.10.2/container.go:299 +0x470 github.com/emicklei/go-restful/v3.(*Container).Dispatch(...) /go/pkg/mod/github.com/emicklei/go-restful/v3@v3.10.2/container.go:204 k8s.io/apiserver/pkg/server.director.ServeHTTP({{0x1d671da?, 0x116b334?}, 0x4005a24ab0?, 0x4000784b60?}, {0x210c8d0, 0x4005a6af40}, 0x4003566100) /go/pkg/mod/k8s.io/apiserver@v0.27.2/pkg/server/handler.go:146 +0x46c k8s.io/apiserver/pkg/endpoints/filterlatency.trackCompleted.func1({0x210c8d0, 0x4005a6af40}, 0x4003566100) /go/pkg/mod/k8s.io/apiserver@v0.27.2/pkg/endpoints/filterlatency/filterlatency.go:110 +0x174 net/http.HandlerFunc.ServeHTTP(0x210d5b8?, {0x210c8d0?, 0x4005a6af40?}, 0x4?) /usr/local/go/src/net/http/server.go:2122 +0x38 k8s.io/apiserver/pkg/endpoints/filters.WithAuthorization.func1({0x210c8d0, 0x4005a6af40}, 0x4003566100) /go/pkg/mod/k8s.io/apiserver@v0.27.2/pkg/endpoints/filters/authorization.go:64 +0x39c net/http.HandlerFunc.ServeHTTP(0x433a3342b2?, {0x210c8d0?, 0x4005a6af40?}, 0x912280?) /usr/local/go/src/net/http/server.go:2122 +0x38 k8s.io/apiserver/pkg/endpoints/filterlatency.trackStarted.func1({0x210c8d0, 0x4005a6af40}, 0x4003566100) /go/pkg/mod/k8s.io/apiserver@v0.27.2/pkg/endpoints/filterlatency/filterlatency.go:84 +0x150 net/http.HandlerFunc.ServeHTTP(0x327c3e0?, {0x210c8d0?, 0x4005a6af40?}, 0x4?) /usr/local/go/src/net/http/server.go:2122 +0x38 k8s.io/apiserver/pkg/server/filters.WithMaxInFlightLimit.func1({0x210c8d0, 0x4005a6af40}, 0x4003566100) /go/pkg/mod/k8s.io/apiserver@v0.27.2/pkg/server/filters/maxinflight.go:197 +0x22c net/http.HandlerFunc.ServeHTTP(0x32dde98?, {0x210c8d0?, 0x4005a6af40?}, 0x12c50d4?) /usr/local/go/src/net/http/server.go:2122 +0x38 k8s.io/apiserver/pkg/endpoints/filterlatency.trackCompleted.func1({0x210c8d0, 0x4005a6af40}, 0x4003566100) /go/pkg/mod/k8s.io/apiserver@v0.27.2/pkg/endpoints/filterlatency/filterlatency.go:110 +0x174 net/http.HandlerFunc.ServeHTTP(0x4007439208?, {0x210c8d0?, 0x4005a6af40?}, 0x163?) /usr/local/go/src/net/http/server.go:2122 +0x38 k8s.io/apiserver/pkg/endpoints/filters.WithImpersonation.func1({0x210c8d0, 0x4005a6af40}, 0x4003566100) /go/pkg/mod/k8s.io/apiserver@v0.27.2/pkg/endpoints/filters/impersonation.go:50 +0x1a4 net/http.HandlerFunc.ServeHTTP(0x20f2498?, {0x210c8d0?, 0x4005a6af40?}, 0xa?) /usr/local/go/src/net/http/server.go:2122 +0x38 k8s.io/apiserver/pkg/endpoints/filterlatency.trackStarted.func1({0x210c8d0, 0x4005a6af40}, 0x4003566100) /go/pkg/mod/k8s.io/apiserver@v0.27.2/pkg/endpoints/filterlatency/filterlatency.go:84 +0x150 net/http.HandlerFunc.ServeHTTP(0x32dde98?, {0x210c8d0?, 0x4005a6af40?}, 0x12c50d4?) /usr/local/go/src/net/http/server.go:2122 +0x38 k8s.io/apiserver/pkg/endpoints/filterlatency.trackCompleted.func1({0x210c8d0, 0x4005a6af40}, 0x4003566100) /go/pkg/mod/k8s.io/apiserver@v0.27.2/pkg/endpoints/filterlatency/filterlatency.go:110 +0x174 net/http.HandlerFunc.ServeHTTP(0x32a7660?, {0x210c8d0?, 0x4005a6af40?}, 0xa?) /usr/local/go/src/net/http/server.go:2122 +0x38 k8s.io/apiserver/pkg/endpoints/filterlatency.trackStarted.func1({0x210c8d0, 0x4005a6af40}, 0x4003566100) /go/pkg/mod/k8s.io/apiserver@v0.27.2/pkg/endpoints/filterlatency/filterlatency.go:84 +0x150 net/http.HandlerFunc.ServeHTTP(0x32dde98?, {0x210c8d0?, 0x4005a6af40?}, 0x12c50d4?) /usr/local/go/src/net/http/server.go:2122 +0x38 k8s.io/apiserver/pkg/endpoints/filterlatency.trackCompleted.func1({0x210c8d0, 0x4005a6af40}, 0x4003566100) /go/pkg/mod/k8s.io/apiserver@v0.27.2/pkg/endpoints/filterlatency/filterlatency.go:110 +0x174 net/http.HandlerFunc.ServeHTTP(0x210d5b8?, {0x210c8d0?, 0x4005a6af40?}, 0x20dd120?) /usr/local/go/src/net/http/server.go:2122 +0x38 k8s.io/apiserver/pkg/endpoints/filters.withAuthentication.func1({0x210c8d0, 0x4005a6af40}, 0x4003566100) /go/pkg/mod/k8s.io/apiserver@v0.27.2/pkg/endpoints/filters/authentication.go:105 +0x504 net/http.HandlerFunc.ServeHTTP(0x210d5b8?, {0x210c8d0?, 0x4005a6af40?}, 0x20e1cc0?) /usr/local/go/src/net/http/server.go:2122 +0x38 k8s.io/apiserver/pkg/endpoints/filterlatency.trackStarted.func1({0x210c8d0, 0x4005a6af40}, 0x400354df00) /go/pkg/mod/k8s.io/apiserver@v0.27.2/pkg/endpoints/filterlatency/filterlatency.go:94 +0x2f4 net/http.HandlerFunc.ServeHTTP(0x4001378780?, {0x210c8d0?, 0x4005a6af40?}, 0x7cf54?) /usr/local/go/src/net/http/server.go:2122 +0x38 k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP.func1() /go/pkg/mod/k8s.io/apiserver@v0.27.2/pkg/server/filters/timeout.go:115 +0x68 created by k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP /go/pkg/mod/k8s.io/apiserver@v0.27.2/pkg/server/filters/timeout.go:101 +0x188 goroutine 1464 [running]: golang.org/x/net/http2.(*serverConn).runHandler.func1() /go/pkg/mod/golang.org/x/net@v0.8.0/http2/server.go:2304 +0x138 panic({0x1903ca0, 0x4003535810}) /usr/local/go/src/runtime/panic.go:884 +0x1f4 k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x40038c8cd0, 0x1, 0x400736cc60?}) /go/pkg/mod/k8s.io/apimachinery@v0.27.2/pkg/util/runtime/runtime.go:56 +0xe0 panic({0x1903ca0, 0x4003535810}) /usr/local/go/src/runtime/panic.go:884 +0x1f4 k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP(0x40046c8db0, {0x210c8d0, 0x4005a6ae80}, 0xdf8475800?) /go/pkg/mod/k8s.io/apiserver@v0.27.2/pkg/server/filters/timeout.go:121 +0x2d4 k8s.io/apiserver/pkg/endpoints/filters.withRequestDeadline.func1({0x210c8d0, 0x4005a6ae80}, 0x400354dd00) /go/pkg/mod/k8s.io/apiserver@v0.27.2/pkg/endpoints/filters/request_deadline.go:100 +0x298 net/http.HandlerFunc.ServeHTTP(0xffff7db645d8?, {0x210c8d0?, 0x4005a6ae80?}, 0x1cc4c?) /usr/local/go/src/net/http/server.go:2122 +0x38 k8s.io/apiserver/pkg/server/filters.withWaitGroup.func1({0x210c8d0, 0x4005a6ae80}, 0x400354dd00) /go/pkg/mod/k8s.io/apiserver@v0.27.2/pkg/server/filters/waitgroup.go:86 +0x18c net/http.HandlerFunc.ServeHTTP(0x210d5b8?, {0x210c8d0?, 0x4005a6ae80?}, 0x20e1cc0?) /usr/local/go/src/net/http/server.go:2122 +0x38 k8s.io/apiserver/pkg/endpoints/filters.WithWarningRecorder.func1({0x210c8d0?, 0x4005a6ae80}, 0x400354dc00) /go/pkg/mod/k8s.io/apiserver@v0.27.2/pkg/endpoints/filters/warning.go:35 +0x180 net/http.HandlerFunc.ServeHTTP(0x1b833a0?, {0x210c8d0?, 0x4005a6ae80?}, 0xd?) /usr/local/go/src/net/http/server.go:2122 +0x38 k8s.io/apiserver/pkg/endpoints/filters.WithCacheControl.func1({0x210c8d0, 0x4005a6ae80}, 0xffff7df188d0?) /go/pkg/mod/k8s.io/apiserver@v0.27.2/pkg/endpoints/filters/cachecontrol.go:31 +0x118 net/http.HandlerFunc.ServeHTTP(0x210e268?, {0x210c8d0?, 0x4005a6ae80?}, 0x20e1cc0?) /usr/local/go/src/net/http/server.go:2122 +0x38 k8s.io/apiserver/pkg/server/httplog.withLogging.func1({0x210cff0, 0x4005d16f40}, 0x400354db00) /go/pkg/mod/k8s.io/apiserver@v0.27.2/pkg/server/httplog/httplog.go:131 +0x28c net/http.HandlerFunc.ServeHTTP(0x210d5b8?, {0x210cff0?, 0x4005d16f40?}, 0x20e1cc0?) /usr/local/go/src/net/http/server.go:2122 +0x38 go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.(*Handler).ServeHTTP(0x40000e48f0, {0x210c8d0?, 0x4005a6ad40}, 0x400354d700) /go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.35.1/handler.go:204 +0xe6c k8s.io/apiserver/pkg/endpoints/filters.WithLatencyTrackers.func1({0x20fed10?, 0x4003a7e028}, 0x400354d600) /go/pkg/mod/k8s.io/apiserver@v0.27.2/pkg/endpoints/filters/webhook_duration.go:57 +0x23c net/http.HandlerFunc.ServeHTTP(0x210d5b8?, {0x20fed10?, 0x4003a7e028?}, 0x20e1cc0?) /usr/local/go/src/net/http/server.go:2122 +0x38 k8s.io/apiserver/pkg/endpoints/filters.WithRequestInfo.func1({0x20fed10, 0x4003a7e028}, 0x400354d500) /go/pkg/mod/k8s.io/apiserver@v0.27.2/pkg/endpoints/filters/requestinfo.go:39 +0x1b8 net/http.HandlerFunc.ServeHTTP(0x210d5b8?, {0x20fed10?, 0x4003a7e028?}, 0x433a2e934c?) /usr/local/go/src/net/http/server.go:2122 +0x38 k8s.io/apiserver/pkg/endpoints/filters.withRequestReceivedTimestampWithClock.func1({0x20fed10, 0x4003a7e028}, 0x400354d400) /go/pkg/mod/k8s.io/apiserver@v0.27.2/pkg/endpoints/filters/request_received_time.go:38 +0x13c net/http.HandlerFunc.ServeHTTP(0x4002c36ab8?, {0x20fed10?, 0x4003a7e028?}, 0xffffa54c0108?) /usr/local/go/src/net/http/server.go:2122 +0x38 k8s.io/apiserver/pkg/endpoints/filters.WithMuxAndDiscoveryComplete.func1({0x20fed10, 0x4003a7e028}, 0x400354d400) /go/pkg/mod/k8s.io/apiserver@v0.27.2/pkg/endpoints/filters/mux_discovery_complete.go:52 +0x180 net/http.HandlerFunc.ServeHTTP(0xffff7dbcece8?, {0x20fed10?, 0x4003a7e028?}, 0x4000844400?) /usr/local/go/src/net/http/server.go:2122 +0x38 k8s.io/apiserver/pkg/server/filters.withPanicRecovery.func1({0x20fed10?, 0x4003a7e028?}, 0x4002f9b101?) /go/pkg/mod/k8s.io/apiserver@v0.27.2/pkg/server/filters/wrap.go:74 +0x9c net/http.HandlerFunc.ServeHTTP(0x1b833a0?, {0x20fed10?, 0x4003a7e028?}, 0x8?) /usr/local/go/src/net/http/server.go:2122 +0x38 k8s.io/apiserver/pkg/endpoints/filters.withAuditInit.func1({0x20fed10, 0x4003a7e028}, 0x400354d300) /go/pkg/mod/k8s.io/apiserver@v0.27.2/pkg/endpoints/filters/audit_init.go:63 +0x240 net/http.HandlerFunc.ServeHTTP(0x1d4a967?, {0x20fed10?, 0x4003a7e028?}, 0x3?) /usr/local/go/src/net/http/server.go:2122 +0x38 k8s.io/apiserver/pkg/server.(*APIServerHandler).ServeHTTP(0x0?, {0x20fed10?, 0x4003a7e028?}, 0x1c7fc?) /go/pkg/mod/k8s.io/apiserver@v0.27.2/pkg/server/handler.go:189 +0x30 net/http.serverHandler.ServeHTTP({0x4000058058?}, {0x20fed10, 0x4003a7e028}, 0x400354d300) /usr/local/go/src/net/http/server.go:2936 +0x2c0 net/http.initALPNRequest.ServeHTTP({{0x210d5b8?, 0x4007340840?}, 0x4000a14a80?, {0x400618e0f0?}}, {0x20fed10, 0x4003a7e028}, 0x400354d300) /usr/local/go/src/net/http/server.go:3545 +0x1d0 golang.org/x/net/http2.(*serverConn).runHandler(0x0?, 0x7cf54?, 0x4007258c00?, 0x0?) /go/pkg/mod/golang.org/x/net@v0.8.0/http2/server.go:2311 +0x78 created by golang.org/x/net/http2.(*serverConn).processHeaders /go/pkg/mod/golang.org/x/net@v0.8.0/http2/server.go:2025 +0x58c I0117 21:06:42.704196 1 httplog.go:132] "HTTP" verb="GET" URI="/healthz" latency="143.919µs" userAgent="kube-probe/1.28" audit-ID="7f1ca763-2e93-4173-a77e-9abbb5ff9af2" srcIP="10.244.0.1:41000" resp=200 I0117 21:06:42.704485 1 httplog.go:132] "HTTP" verb="GET" URI="/healthz" latency="67.584µs" userAgent="kube-probe/1.28" audit-ID="84404544-fe98-4b6e-bd6b-49b747cb8af2" srcIP="10.244.0.1:40990" resp=200 I0117 21:06:45.829907 1 httplog.go:132] "HTTP" verb="GET" URI="/apis/custom.metrics.k8s.io/v1beta1/namespaces/kong/services/%2A/kong_upstream_latency_ms" latency="4.124525ms" userAgent="kubectl/v1.29.0 (darwin/arm64) kubernetes/3f7a50f" audit-ID="2bb83135-de14-4236-9b5c-e512d11625d1" srcIP="10.244.0.1:38584" resp=404 I0117 21:06:48.453310 1 httplog.go:132] "HTTP" verb="GET" URI="/apis/custom.metrics.k8s.io/v1beta1/namespaces/kong/pods/%2A/kong_upstream_latency_ms" latency="7.282783ms" userAgent="kubectl/v1.29.0 (darwin/arm64) kubernetes/3f7a50f" audit-ID="68f2e92d-dac0-4a71-b7bf-84be159f4c12" srcIP="10.244.0.1:38584" resp=404
What did you expect to happen?:
No panic.
panic
Please provide the prometheus-adapter config:
Anything else we need to know?:
Environment:
registry.k8s.io/prometheus-adapter/prometheus-adapter:v0.11.2
kubectl version
/triage accepted /assign
What happened?:
When playing around with config and querying a namespace metric I got a panic:
What did you expect to happen?:
No
panic
.Please provide the prometheus-adapter config:
prometheus-adapter config
``` prometheus: url: http://prometheus-kube-prometheus-prometheus.kong.svc rules: custom: - seriesQuery: '{__name__=~"^kong_upstream_latency_ms_"}' resources: template: "<<.Resource>>." overrides: namespace: {resource: "namespace"} pod: {resource: "pod"} name: as: "kong_upstream_latency_ms" metricsQuery: |- sum(rate(kong_upstream_latency_ms_sum{route=~".*httproute.*"}[5m])) by (exported_service) / sum(rate(kong_upstream_latency_ms_count{route=~".*httproute.*"}[5m])) by (exported_service) ```Anything else we need to know?:
Environment:
registry.k8s.io/prometheus-adapter/prometheus-adapter:v0.11.2
kubectl version
): 1.28.0