Open pedropaulopereiraamarante opened 2 months ago
/kind support /triage accepted
any updates? ='(
If you install by helm upgrade --install prometheus-adapter prometheus-community/prometheus-adapter
then it will show only custom metrics but if you enable external rules like helm upgrade --install prometheus-adapter prometheus-community/prometheus-adapter --set-json rules.external=[{}]
then you will see external metrics as well.
What happened?: I would like to be able to get the metrics from both APIs.
What did you expect to happen?: I should be able to get metrics from both APIs (external/custom). I've tried everything, and only the custom API is working. Please provide the prometheus-adapter config:
I have tried all possible configurations, none result in the external.metrics API working.
ex: 2 not worked too.**
Please provide the HPA resource used for autoscaling:
HPA yaml
``` apiVersion: autoscaling/v2 kind: HorizontalPodAutoscaler metadata: name: growth-service-campaign-notification-sidekiq-hpa namespace: growth-service spec: scaleTargetRef: apiVersion: apps/v1 kind: Deployment name: growth-service-campaign-notification-sidekiq minReplicas: 5 maxReplicas: 21 metrics: - type: External external: metric: name: sidekiq_enqueued_jobs target: type: AverageValue averageValue: 900k ```Please provide the HPA status:
Please provide the prometheus-adapter logs with -v=6 around the time the issue happened:
prometheus-adapter logs
`I0618 20:52:33.477517 1 adapter.go:114] successfully using in-cluster auth I0618 20:52:33.484906 1 round_trippers.go:553] GET https://10.100.0.1:443/api?timeout=32s 200 OK in 6 milliseconds I0618 20:52:33.488166 1 round_trippers.go:553] GET https://10.100.0.1:443/apis?timeout=32s 200 OK in 2 milliseconds I0618 20:52:33.504567 1 api.go:88] GET http://prometheus.monitoring.svc:9090/api/v1/series?match%5B%5D=sidekiq_queue_enqueued_jobs%7Bname%3D%22campaign-notification-events%22%2Cnamespace%21%3D%22%22%2Cpod%21%3D%22%22%7D&start=1718743893.491 200 OK I0618 20:52:33.794448 1 serving.go:342] Generated self-signed cert (/tmp/cert/apiserver.crt, /tmp/cert/apiserver.key) I0618 20:52:33.795538 1 dynamic_serving_content.go:113] "Loaded a new cert/key pair" name="serving-cert::/tmp/cert/apiserver.crt::/tmp/cert/apiserver.key" I0618 20:52:34.366098 1 round_trippers.go:553] GET https://10.100.0.1:443/api/v1/namespaces/kube-system/configmaps/extension-apiserver-authentication 200 OK in 4 milliseconds I0618 20:52:34.366900 1 requestheader_controller.go:244] Loaded a new request header values for RequestHeaderAuthRequestController I0618 20:52:34.367302 1 maxinflight.go:140] "Initialized nonMutatingChan" len=400 I0618 20:52:34.367360 1 maxinflight.go:146] "Initialized mutatingChan" len=200 I0618 20:52:34.367407 1 timing_ratio_histogram.go:203] "TimingRatioHistogramVec.NewForLabelValuesSafe hit the inefficient case" fqName="apiserver_flowcontrol_read_vs_write_current_requests" labelValues=[executing readOnly] I0618 20:52:34.367451 1 timing_ratio_histogram.go:203] "TimingRatioHistogramVec.NewForLabelValuesSafe hit the inefficient case" fqName="apiserver_flowcontrol_read_vs_write_current_requests" labelValues=[executing mutating] I0618 20:52:34.367488 1 maxinflight.go:117] "Set denominator for readonly requests" limit=400 I0618 20:52:34.367512 1 maxinflight.go:121] "Set denominator for mutating requests" limit=200 I0618 20:52:34.369387 1 config.go:814] Not requested to run hook priority-and-fairness-config-consumer I0618 20:52:34.374917 1 healthz.go:176] Installing health checkers for (/healthz): "ping","log","poststarthook/max-in-flight-filter","poststarthook/storage-object-count-tracker-hook" I0618 20:52:34.375229 1 healthz.go:176] Installing health checkers for (/livez): "ping","log","poststarthook/max-in-flight-filter","poststarthook/storage-object-count-tracker-hook" I0618 20:52:34.375483 1 healthz.go:176] Installing health checkers for (/readyz): "ping","log","poststarthook/max-in-flight-filter","poststarthook/storage-object-count-tracker-hook","shutdown" I0618 20:52:34.375965 1 genericapiserver.go:524] MuxAndDiscoveryComplete has all endpoints registered and discovery information is complete I0618 20:52:34.380054 1 round_trippers.go:553] GET https://10.100.0.1:443/api/v1/namespaces/kube-system/configmaps/extension-apiserver-authentication 200 OK in 4 milliseconds I0618 20:52:34.384706 1 tlsconfig.go:200] "Loaded serving cert" certName="serving-cert::/tmp/cert/apiserver.crt::/tmp/cert/apiserver.key" certDetail="\"localhost@1718743953\" [serving] validServingFor=[127.0.0.1,localhost,localhost] issuer=\"localhost-ca@1718743953\" (2024-06-18 19:52:33 +0000 UTC to 2025-06-18 19:52:33 +0000 UTC (now=2024-06-18 20:52:34.384668328 +0000 UTC))" I0618 20:52:34.385141 1 named_certificates.go:53] "Loaded SNI cert" index=0 certName="self-signed loopback" certDetail="\"apiserver-loopback-client@1718743954\" [serving] validServingFor=[apiserver-loopback-client] issuer=\"apiserver-loopback-client-ca@1718743953\" (2024-06-18 19:52:33 +0000 UTC to 2025-06-18 19:52:33 +0000 UTC (now=2024-06-18 20:52:34.385114205 +0000 UTC))" I0618 20:52:34.385335 1 secure_serving.go:210] Serving securely on [::]:6443 I0618 20:52:34.385437 1 genericapiserver.go:672] [graceful-termination] waiting for shutdown to be initiated I0618 20:52:34.385550 1 requestheader_controller.go:169] Starting RequestHeaderAuthRequestController I0618 20:52:34.385633 1 shared_informer.go:311] Waiting for caches to sync for RequestHeaderAuthRequestController I0618 20:52:34.385753 1 dynamic_serving_content.go:132] "Starting controller" name="serving-cert::/tmp/cert/apiserver.crt::/tmp/cert/apiserver.key" I0618 20:52:34.385955 1 tlsconfig.go:240] "Starting DynamicServingCertificateController" I0618 20:52:34.386206 1 configmap_cafile_content.go:202] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::client-ca-file" I0618 20:52:34.386229 1 shared_informer.go:311] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file I0618 20:52:34.386364 1 configmap_cafile_content.go:202] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file" I0618 20:52:34.386380 1 shared_informer.go:311] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file I0618 20:52:34.386641 1 reflector.go:287] Starting reflector *v1.ConfigMap (12h0m0s) from pkg/mod/k8s.io/client-go@v0.27.2/tools/cache/reflector.go:231 I0618 20:52:34.386658 1 reflector.go:323] Listing and watching *v1.ConfigMap from pkg/mod/k8s.io/client-go@v0.27.2/tools/cache/reflector.go:231 I0618 20:52:34.387699 1 reflector.go:287] Starting reflector *v1.ConfigMap (12h0m0s) from pkg/mod/k8s.io/client-go@v0.27.2/tools/cache/reflector.go:231 I0618 20:52:34.387719 1 reflector.go:323] Listing and watching *v1.ConfigMap from pkg/mod/k8s.io/client-go@v0.27.2/tools/cache/reflector.go:231 I0618 20:52:34.387921 1 reflector.go:287] Starting reflector *v1.ConfigMap (12h0m0s) from pkg/mod/k8s.io/client-go@v0.27.2/tools/cache/reflector.go:231 I0618 20:52:34.388082 1 reflector.go:323] Listing and watching *v1.ConfigMap from pkg/mod/k8s.io/client-go@v0.27.2/tools/cache/reflector.go:231 I0618 20:52:34.388641 1 round_trippers.go:553] GET https://10.100.0.1:443/api/v1/namespaces/kube-system/configmaps?fieldSelector=metadata.name%3Dextension-apiserver-authentication&limit=500&resourceVersion=0 200 OK in 1 milliseconds I0618 20:52:34.389215 1 round_trippers.go:553] GET https://10.100.0.1:443/api/v1/namespaces/kube-system/configmaps?fieldSelector=metadata.name%3Dextension-apiserver-authentication&limit=500&resourceVersion=0 200 OK in 0 milliseconds I0618 20:52:34.391774 1 round_trippers.go:553] GET https://10.100.0.1:443/api/v1/namespaces/kube-system/configmaps?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dextension-apiserver-authentication&resourceVersion=14273812&timeout=8m21s&timeoutSeconds=501&watch=true 200 OK in 1 milliseconds I0618 20:52:34.392861 1 round_trippers.go:553] GET https://10.100.0.1:443/api/v1/namespaces/kube-system/configmaps?fieldSelector=metadata.name%3Dextension-apiserver-authentication&limit=500&resourceVersion=0 200 OK in 2 milliseconds I0618 20:52:34.394447 1 round_trippers.go:553] GET https://10.100.0.1:443/api/v1/namespaces/kube-system/configmaps?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dextension-apiserver-authentication&resourceVersion=14273812&timeout=9m49s&timeoutSeconds=589&watch=true 200 OK in 0 milliseconds I0618 20:52:34.396393 1 round_trippers.go:553] GET https://10.100.0.1:443/api/v1/namespaces/kube-system/configmaps?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dextension-apiserver-authentication&resourceVersion=14273812&timeout=6m55s&timeoutSeconds=415&watch=true 200 OK in 0 milliseconds I0618 20:52:34.486580 1 shared_informer.go:341] caches populated I0618 20:52:34.486598 1 shared_informer.go:341] caches populated I0618 20:52:34.486603 1 shared_informer.go:318] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file I0618 20:52:34.486615 1 shared_informer.go:318] Caches are synced for RequestHeaderAuthRequestController I0618 20:52:34.486748 1 shared_informer.go:341] caches populated I0618 20:52:34.486758 1 shared_informer.go:318] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file I0618 20:52:34.486927 1 tlsconfig.go:178] "Loaded client CA" index=0 certName="client-ca::kube-system::extension-apiserver-authentication::client-ca-file,client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file" certDetail="\"front-proxy-ca\" [] validServingFor=[front-proxy-ca] issuer=\"Anything else we need to know?:
Environment:
kubectl version
): $k version Client Version: v1.30.1 Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3 Server Version: v1.28.9-eks-036c24b