search

kubernetes-sigs / prometheus-adapter

An implementation of the custom.metrics.k8s.io API using Prometheus
Apache License 2.0
1.9k stars 551 forks source link

Unable to query metric for process_cpu_seconds_total #582

Open sarasensible opened 1 year ago

sarasensible commented 1 year ago

What happened?: I am trying to create a custom metric for process_cpu_seconds_total. I see the metric in the api raw response however I'm unable to query it:

    {
      "name": "pods/process_cpu_seconds_total",
      "singularName": "",
      "namespaced": true,
      "kind": "MetricValueList",
      "verbs": [
        "get"
      ]
    },
kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/weather/pods/*/process_cpu_seconds_total"
Error from server (InternalError): Internal error occurred: unable to fetch metrics

What did you expect to happen?: I expected to be able to query my metric.

Please provide the prometheus-adapter config: prometheus-adapter config:

- seriesQuery: 'process_cpu_seconds_total{namespace!="",pod!=""}'
    resources:
      overrides:
        namespace: {resource: "namespace"}
        pod: {resource: "pod"}
    metricsQuery: rate(<<.Series>>{<<.LabelMatchers>>}[1m]) by (<<.GroupBy>>)

Please provide the HPA resource used for autoscaling: N/A

Please provide the HPA status: N/A

Please provide the prometheus-adapter logs with -v=6 around the time the issue happened:

prometheus-adapter logs ``` I0510 19:54:39.977398 1 api.go:76] GET http://prometheus-stack-1-kube-pr-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=rate%28process_cpu_seconds_total%7Bnamespace%3D%22weather%22%2Cpod%3D%22%2A%22%7D%5B1m%5D%29+by+%28pod%29&time=1683748479.975&timeout= 400 Bad Request E0510 19:54:39.977516 1 provider.go:150] unable to fetch metrics from prometheus: bad_data: invalid parameter "query": 1:97: parse error: unexpected I0510 19:54:39.978042 1 httplog.go:104] "HTTP" verb="GET" URI="/apis/custom.metrics.k8s.io/v1beta1/namespaces/weather/pods/*/process_cpu_seconds_total" latency="2.865805ms" userAgent="kubectl/v1.26.2 (darwin/amd64) kubernetes/fc04e73" audit-ID="d3885e44-d4c3-4f2a-905d-6de40882dffa" srcIP="10.0.21.111:34356" resp=500 statusStack="\ngoroutine 8441 [running]:\nk8s.io/apiserver/pkg/server/httplog.(*respLogger).recordStatus(0xc01a1b8200, 0x1f4)\n\t/go/pkg/mod/k8s.io/apiserver@v0.22.0/pkg/server/httplog/httplog.go:274 +0xcf\nk8s.io/apiserver/pkg/server/httplog.(*respLogger).WriteHeader(0xc01a1b8200, 0x1f4)\n\t/go/pkg/mod/k8s.io/apiserver@v0.22.0/pkg/server/httplog/httplog.go:252 +0x35\nk8s.io/apiserver/pkg/server/filters.(*baseTimeoutWriter).WriteHeader(0xc01906d220, 0x1f4)\n\t/go/pkg/mod/k8s.io/apiserver@v0.22.0/pkg/server/filters/timeout.go:227 +0xb2\nk8s.io/apiserver/pkg/endpoints/metrics.(*ResponseWriterDelegator).WriteHeader(0xc01a19be60, 0x1f4)\n\t/go/pkg/mod/k8s.io/apiserver@v0.22.0/pkg/endpoints/metrics/metrics.go:603 +0x45\nk8s.io/apiserver/pkg/endpoints/handlers/responsewriters.(*deferredResponseWriter).Write(0xc01a1bd980, 0xc023ee6000, 0xe3, 0x132ab1, 0x0, 0x0, 0x0)\n\t/go/pkg/mod/k8s.io/apiserver@v0.22.0/pkg/endpoints/handlers/responsewriters/writers.go:228 +0x2fd\nencoding/json.(*Encoder).Encode(0xc0009a95b8, 0x206cfc0, 0xc01a1d4320, 0x7c13fa, 0x20be562)\n\t/usr/local/go/src/encoding/json/stream.go:231 +0x1df\nk8s.io/apimachinery/pkg/runtime/serializer/json.(*Serializer).doEncode(0xc0000a33b0, 0x2386eb0, 0xc01a1d4320, 0x237aba0, 0xc01a1bd980, 0x0, 0x0)\n\t/go/pkg/mod/k8s.io/apimachinery@v0.22.0/pkg/runtime/serializer/json/json.go:327 +0x2e9\nk8s.io/apimachinery/pkg/runtime/serializer/json.(*Serializer).Encode(0xc0000a33b0, 0x2386eb0, 0xc01a1d4320, 0x237aba0, 0xc01a1bd980, 0x1beadad, 0x6)\n\t/go/pkg/mod/k8s.io/apimachinery@v0.22.0/pkg/runtime/serializer/json/json.go:301 +0x169\nk8s.io/apimachinery/pkg/runtime/serializer/versioning.(*codec).doEncode(0xc01a1d43c0, 0x2386eb0, 0xc01a1d4320, 0x237aba0, 0xc01a1bd980, 0x0, 0x0)\n\t/go/pkg/mod/k8s.io/apimachinery@v0.22.0/pkg/runtime/serializer/versioning/versioning.go:228 +0x3b6\nk8s.io/apimachinery/pkg/runtime/serializer/versioning.(*codec).Encode(0xc01a1d43c0, 0x2386eb0, 0xc01a1d4320, 0x237aba0, 0xc01a1bd980, 0xc000403c80, 0x3)\n\t/go/pkg/mod/k8s.io/apimachinery@v0.22.0/pkg/runtime/serializer/versioning/versioning.go:184 +0x170\nk8s.io/apiserver/pkg/endpoints/handlers/responsewriters.SerializeObject(0x20ce8b4, 0x10, 0x7f7f2fcf8a58, 0xc01a1d43c0, 0x23b77a0, 0xc0040e22c0, 0xc01a1d6200, 0x1f4, 0x2386eb0, 0xc01a1d4320)\n\t/go/pkg/mod/k8s.io/apiserver@v0.22.0/pkg/endpoints/handlers/responsewriters/writers.go:106 +0x457\nk8s.io/apiserver/pkg/endpoints/handlers/responsewriters.WriteObjectNegotiated(0x23b9330, 0xc00038f340, 0x23b94b0, 0x31c7030, 0x20d61f9, 0x15, 0x20c3567, 0x7, 0x23b77a0, 0xc0040e22c0, ...)\n\t/go/pkg/mod/k8s.io/apiserver@v0.22.0/pkg/endpoints/handlers/responsewriters/writers.go:275 +0x5cd\nk8s.io/apiserver/pkg/endpoints/handlers/responsewriters.ErrorNegotiated(0x237a6c0, 0xc01a1d4280, 0x23b9330, 0xc00038f340, 0x20d61f9, 0x15, 0x20c3567, 0x7, 0x23b77a0, 0xc0040e22c0, ...)\n\t/go/pkg/mod/k8s.io/apiserver@v0.22.0/pkg/endpoints/handlers/responsewriters/writers.go:294 +0x16f\nsigs.k8s.io/custom-metrics-apiserver/pkg/apiserver/endpoints/handlers.writeError(...)\n\t/go/pkg/mod/sigs.k8s.io/custom-metrics-apiserver@v1.22.0/pkg/apiserver/endpoints/handlers/rest.go:33\nsigs.k8s.io/custom-metrics-apiserver/pkg/apiserver/endpoints/handlers.ListResourceWithOptions.func1(0x23b77a0, 0xc0040e22c0, 0xc01a1d6200)\n\t/go/pkg/mod/sigs.k8s.io/custom-metrics-apiserver@v1.22.0/pkg/apiserver/endpoints/handlers/get.go:118 +0xce8\nsigs.k8s.io/custom-metrics-apiserver/pkg/apiserver/installer.restfulListResourceWithOptions.func1(0xc01a19bdd0, 0xc000195a40)\n\t/go/pkg/mod/sigs.k8s.io/custom-metrics-apiserver@v1.22.0/pkg/apiserver/installer/installer.go:297 +0xed\nk8s.io/apiserver/pkg/endpoints/metrics.InstrumentRouteFunc.func1(0xc01a19bdd0, 0xc000195a40)\n\t/go/pkg/mod/k8s.io/apiserver@v0.22.0/pkg/endpoints/metrics/metrics.go:484 +0x2d5\ngithub.com/emicklei/go-restful.(*Container).dispatch(0xc00030dc20, 0x7f7f2fcf86a8, 0xc0040e22a8, 0xc01a1d6200)\n\t/go/pkg/mod/github.com/emicklei/go-restful@v2.15.0+incompatible/container.go:294 +0x65a\ngithub.com/emicklei/go-restful.(*Container).Dispatch(...)\n\t/go/pkg/mod/github.com/emicklei/go-restful@v2.15.0+incompatible/container.go:204\nk8s.io/apiserver/pkg/server.director.ServeHTTP(0x20ddf36, 0x1a, 0xc00030dc20, 0xc00013b340, 0x7f7f2fcf86a8, 0xc0040e22a8, 0xc01a1d6200)\n\t/go/pkg/mod/k8s.io/apiserver@v0.22.0/pkg/server/handler.go:146 +0x63e\nk8s.io/apiserver/pkg/endpoints/filterlatency.trackCompleted.func1(0x7f7f2fcf86a8, 0xc0040e22a8, 0xc01a1d6200)\n\t/go/pkg/mod/k8s.io/apiserver@v0.22.0/pkg/endpoints/filterlatency/filterlatency.go:103 +0x193\nnet/http.HandlerFunc.ServeHTTP(0xc000676360, 0x7f7f2fcf86a8, 0xc0040e22a8, 0xc01a1d6200)\n\t/usr/local/go/src/net/http/server.go:2069 +0x44\nk8s.io/apiserver/pkg/endpoints/filters.WithAuthorization.func1(0x7f7f2fcf86a8, 0xc0040e22a8, 0xc01a1d6200)\n\t/go/pkg/mod/k8s.io/apiserver@v0.22.0/pkg/endpoints/filters/authorization.go:64 +0x603\nnet/http.HandlerFunc.ServeHTTP(0xc00038eac0, 0x7f7f2fcf86a8, 0xc0040e22a8, 0xc01a1d6200)\n\t/usr/local/go/src/net/http/server.go:2069 +0x44\nk8s.io/apiserver/pkg/endpoints/filterlatency.trackStarted.func1(0x7f7f2fcf86a8, 0xc0040e22a8, 0xc01a1d6200)\n\t/go/pkg/mod/k8s.io/apiserver@v0.22.0/pkg/endpoints/filterlatency/filterlatency.go:79 +0x186\nnet/http.HandlerFunc.ServeHTTP(0xc00038eb00, 0x7f7f2fcf86a8, 0xc0040e22a8, 0xc01a1d6200)\n\t/usr/local/go/src/net/http/server.go:2069 +0x44\nk8s.io/apiserver/pkg/server/filters.WithMaxInFlightLimit.func1(0x7f7f2fcf86a8, 0xc0040e22a8, 0xc01a1d6200)\n\t/go/pkg/mod/k8s.io/apiserver@v0.22.0/pkg/server/filters/maxinflight.go:187 +0x676\nnet/http.HandlerFunc.ServeHTTP(0xc000676390, 0x7f7f2fcf86a8, 0xc0040e22a8, 0xc01a1d6200)\n\t/usr/local/go/src/net/http/server.go:2069 +0x44\nk8s.io/apiserver/pkg/endpoints/filterlatency.trackCompleted.func1(0x7f7f2fcf86a8, 0xc0040e22a8, 0xc01a1d6200)\n\t/go/pkg/mod/k8s.io/apiserver@v0.22.0/pkg/endpoints/filterlatency/filterlatency.go:103 +0x193\nnet/http.HandlerFunc.ServeHTTP(0xc0006763c0, 0x7f7f2fcf86a8, 0xc0040e22a8, 0xc01a1d6200)\n\t/usr/local/go/src/net/http/server.go:2069 +0x44\nk8s.io/apiserver/pkg/endpoints/filters.WithImpersonation.func1(0x7f7f2fcf86a8, 0xc0040e22a8, 0xc01a1d6200)\n\t/go/pkg/mod/k8s.io/apiserver@v0.22.0/pkg/endpoints/filters/impersonation.go:50 +0x274d\nnet/http.HandlerFunc.ServeHTTP(0xc00038eb40, 0x7f7f2fcf86a8, 0xc0040e22a8, 0xc01a1d6200)\n\t/usr/local/go/src/net/http/server.go:2069 +0x44\nk8s.io/apiserver/pkg/endpoints/filterlatency.trackStarted.func1(0x7f7f2fcf86a8, 0xc0040e22a8, 0xc01a1d6200)\n\t/go/pkg/mod/k8s.io/apiserver@v0.22.0/pkg/endpoints/filterlatency/filterlatency.go:79 +0x186\nnet/http.HandlerFunc.ServeHTTP(0xc00038eb80, 0x7f7f2fcf86a8, 0xc0040e22a8, 0xc01a1d6200)\n\t/usr/local/go/src/net/http/server.go:2069 +0x44\nk8s.io/apiserver/pkg/endpoints/filterlatency.trackCompleted.func1(0x7f7f2fcf86a8, 0xc0040e22a8, 0xc01a1d6200)\n\t/go/pkg/mod/k8s.io/apiserver@v0.22.0/pkg/endpoints/filterlatency/filterlatency.go:103 +0x193\nnet/http.HandlerFunc.ServeHTTP(0xc0006763f0, 0x7f7f2fcf86a8, 0xc0040e22a8, 0xc01a1d6200)\n\t/usr/local/go/src/net/http/server.go:2069 +0x44\nk8s.io/apiserver/pkg/endpoints/filterlatency.trackStarted.func1(0x7f7f2fcf86a8, 0xc0040e22a8, 0xc01a1d6200)\n\t/go/pkg/mod/k8s.io/apiserver@v0.22.0/pkg/endpoints/filterlatency/filterlatency.go:79 +0x186\nnet/http.HandlerFunc.ServeHTTP(0xc00038ebc0, 0x7f7f2fcf86a8, 0xc0040e22a8, 0xc01a1d6200)\n\t/usr/local/go/src/net/http/server.go:2069 +0x44\nk8s.io/apiserver/pkg/endpoints/filterlatency.trackCompleted.func1(0x7f7f2fcf86a8, 0xc0040e22a8, 0xc01a1d6200)\n\t/go/pkg/mod/k8s.io/apiserver@v0.22.0/pkg/endpoints/filterlatency/filterlatency.go:103 +0x193\nnet/http.HandlerFunc.ServeHTTP(0xc000676450, 0x7f7f2fcf86a8, 0xc0040e22a8, 0xc01a1d6200)\n\t/usr/local/go/src/net/http/server.go:2069 +0x44\nk8s.io/apiserver/pkg/endpoints/filters.withAuthentication.func1(0x7f7f2fcf86a8, 0xc0040e22a8, 0xc01a1d6200)\n\t/go/pkg/mod/k8s.io/apiserver@v0.22.0/pkg/endpoints/filters/authentication.go:80 +0x75c\nnet/http.HandlerFunc.ServeHTTP(0xc0004ed440, 0x7f7f2fcf86a8, 0xc0040e22a8, 0xc01a1d6100)\n\t/usr/local/go/src/net/http/server.go:2069 +0x44\nk8s.io/apiserver/pkg/endpoints/filterlatency.trackStarted.func1(0x7f7f2fcf86a8, 0xc0040e22a8, 0xc01a1d6000)\n\t/go/pkg/mod/k8s.io/apiserver@v0.22.0/pkg/endpoints/filterlatency/filterlatency.go:88 +0x38c\nnet/http.HandlerFunc.ServeHTTP(0xc00038ec00, 0x7f7f2fcf86a8, 0xc0040e22a8, 0xc01a1d6000)\n\t/usr/local/go/src/net/http/server.go:2069 +0x44\nk8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP.func1(0xc019cfd9e0, 0xc000117428, 0x23bc678, 0xc0040e22a8, 0xc01a1d6000)\n\t/go/pkg/mod/k8s.io/apiserver@v0.22.0/pkg/server/filters/timeout.go:108 +0xb8\ncreated by k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP\n\t/go/pkg/mod/k8s.io/apiserver@v0.22.0/pkg/server/filters/timeout.go:94 +0x1fa\n" addedInfo="\nlogging error output: \"{\\\"kind\\\":\\\"Status\\\",\\\"apiVersion\\\":\\\"v1\\\",\\\"metadata\\\":{},\\\"status\\\":\\\"Failure\\\",\\\"message\\\":\\\"Internal error occurred: unable to fetch metrics\\\",\\\"reason\\\":\\\"InternalError\\\",\\\"details\\\":{\\\"causes\\\":[{\\\"message\\\":\\\"unable to fetch metrics\\\"}]},\\\"code\\\":500}\\n\"\n" ```

Anything else we need to know?: I tried playing around with the metric configuration by using kubernetes_namespace/kubernetes_pod_name and just pod_name in the overrides however this only resulted in the metric not showing up in the api query at all.

Environment:

sarasensible commented 1 year ago

For some context I have been using metrics-server for cpu utilization however we are seeing behavior that the cpu reported by the HPA is much lower than that reported by Prometheus. As a result I am looking to get the Prometheus reported value directly into the HPA via a custom metric.

dgrisonnet commented 1 year ago

/kind support /triage accepted

dgrisonnet commented 1 year ago

/assign

k8s-triage-robot commented 2 months ago

This issue has not been updated in over 1 year, and should be re-triaged.

You can:

For more details on the triage process, see https://www.kubernetes.dev/docs/guide/issue-triage/

/remove-triage accepted