dashpole
commented
6 months ago /assign @dgrisonnet /triage accepted
Open jibinrajck opened 6 months ago
dashpole
commented
6 months ago /assign @dgrisonnet /triage accepted
kariya-mitsuru
commented
6 months ago Hi, I also encountered a similar problem. (I referred to this, so strictly speaking it's a bit different.)
I think you are using this ConfigMap. If so, node_cpu_usage_seconds_total and node_memory_working_set_bytes used there are metrics exposed by the kubelet's /metrics/resource endpoint, that is relatively new.
Is this endpoint scraped by Prometheus? If not, NodeMetrics cannot be generated.
On the other hand, it seems that PodMetrics has been obtained, so I think that /metrics/cadvisor has already been scraped by Prometheus. However, if both /metrics/cadvisor and /metrics/resource are scraped, this ConfigMap will generate incorrect PodMetrics values.
This is because container_cpu_usage_seconds_total and container_memory_working_set_bytes used there are exposed by both endpoints, resulting in double counting. Therefore, you will need to take measures such as assigning a metrics_path label using relabel_configs and narrowing it down to one.
As an alternative solution to avoid using /metrics/resource, you could follow the approach mentioned here(I used it). However, in that case, you'll need Prometheus Node Exporter, and you'll also need to assign node names to node label using relabel_configs.
ganchkal
What happened?: kubectl top nodes not giving proper response. Error out with - error: metrics not available yet
What did you expect to happen?: Should return proper response
Please provide the prometheus-adapter config:
prometheus-adapter config
``` apiVersion: apps/v1 kind: Deployment metadata: annotations: deployment.kubernetes.io/revision: "1" creationTimestamp: "2024-02-16T02:52:24Z" generation: 1 labels: app.kubernetes.io/component: metrics-adapter app.kubernetes.io/instance: pipeline-monitoring-b9cb893b app.kubernetes.io/name: prometheus-adapter app.kubernetes.io/part-of: kube-prometheus app.kubernetes.io/version: 0.11.2 name: prometheus-adapter namespace: monitoring-system resourceVersion: "40411" uid: fe95f92d-788d-494b-8579-06dda771455c spec: progressDeadlineSeconds: 600 replicas: 2 revisionHistoryLimit: 10 selector: matchLabels: app.kubernetes.io/component: metrics-adapter app.kubernetes.io/name: prometheus-adapter app.kubernetes.io/part-of: kube-prometheus strategy: rollingUpdate: maxSurge: 1 maxUnavailable: 1 type: RollingUpdate template: metadata: annotations: checksum.config/md5: 3b1ebf7df0232d1675896f67b66373db creationTimestamp: null labels: app.kubernetes.io/component: metrics-adapter app.kubernetes.io/name: prometheus-adapter app.kubernetes.io/part-of: kube-prometheus app.kubernetes.io/version: 0.11.2 spec: affinity: podAntiAffinity: preferredDuringSchedulingIgnoredDuringExecution: - podAffinityTerm: labelSelector: matchLabels: app.kubernetes.io/component: metrics-adapter app.kubernetes.io/name: prometheus-adapter app.kubernetes.io/part-of: kube-prometheus namespaces: - monitoring-system topologyKey: kubernetes.io/hostname weight: 100 automountServiceAccountToken: true containers: - args: - --cert-dir=/var/run/serving-cert - --config=/etc/adapter/config.yaml - --metrics-relist-interval=1m - --prometheus-url=http://prometheus-k8s.monitoring-system.svc:9090/ - --secure-port=6443 - --tls-cipher-suites= XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX image: registry.k8s.io/prometheus-adapter/prometheus-adapter:v0.11.2 imagePullPolicy: IfNotPresent livenessProbe: failureThreshold: 5 httpGet: path: /livez port: https scheme: HTTPS periodSeconds: 5 successThreshold: 1 timeoutSeconds: 1 name: prometheus-adapter ports: - containerPort: 6443 name: https protocol: TCP readinessProbe: failureThreshold: 5 httpGet: path: /readyz port: https scheme: HTTPS periodSeconds: 5 successThreshold: 1 timeoutSeconds: 1 resources: limits: cpu: 250m memory: 180Mi requests: cpu: 102m memory: 180Mi securityContext: allowPrivilegeEscalation: false capabilities: drop: - ALL readOnlyRootFilesystem: true runAsNonRoot: true seccompProfile: type: RuntimeDefault startupProbe: failureThreshold: 18 httpGet: path: /livez port: https scheme: HTTPS periodSeconds: 10 successThreshold: 1 timeoutSeconds: 1 terminationMessagePath: /dev/termination-log terminationMessagePolicy: File volumeMounts: - mountPath: /tmp name: tmpfs - mountPath: /var/run/serving-cert name: volume-serving-cert - mountPath: /etc/adapter name: config dnsPolicy: ClusterFirst nodeSelector: kubernetes.io/os: linux restartPolicy: Always schedulerName: default-scheduler securityContext: {} serviceAccount: prometheus-adapter serviceAccountName: prometheus-adapter terminationGracePeriodSeconds: 30 volumes: - emptyDir: {} name: tmpfs - emptyDir: {} name: volume-serving-cert - configMap: defaultMode: 420 name: adapter-config name: config status: availableReplicas: 2 conditions: - lastTransitionTime: "2024-02-16T02:52:24Z" lastUpdateTime: "2024-02-16T02:54:10Z" message: ReplicaSet "prometheus-adapter-fc7bc9c4d" has successfully progressed. reason: NewReplicaSetAvailable status: "True" type: Progressing - lastTransitionTime: "2024-02-16T03:22:06Z" lastUpdateTime: "2024-02-16T03:22:06Z" message: Deployment has minimum availability. reason: MinimumReplicasAvailable status: "True" type: Available observedGeneration: 1 readyReplicas: 2 replicas: 2 updatedReplicas: 2 ```Please provide the HPA resource used for autoscaling:
HPA yaml
Please provide the HPA status:
Please provide the prometheus-adapter logs with -v=6 around the time the issue happened:
prometheus-adapter logs
``` I0215 04:38:57.171364 1 handler.go:143] prometheus-metrics-adapter: GET "/apis/metrics.k8s.io/v1beta1/nodes" satisfied by gorestful with webservice /apis/metrics.k8s.io/v1beta1 I0215 04:38:57.174856 1 api.go:88] GET http://prometheus-k8s.monitoring-system.svc:9090/api/v1/query?query=sum+by+%28node%29+%28%0A++1+-+irate%28%0A++++node_cpu_seconds_total%7Bmode%3D%22idle%22%7D%5B60s%5D%0A++%29%0A++%2A+on%28namespace%2C+pod%29+group_left%28node%29+%28%0A++++node_namespace_pod%3Akube_pod _info%3A%7Bnode%3D~%22ip-xx-xxx-xxx-xxx.ap-southeast-2.compute.internal%7Cip-xx-xxx-xxx-xxx.ap-southeast-2.compute.internal%7Cip-xx-xxx-xxx-xxx.ap-southeast-2.compute.internal%7Cip-xx-xxx-xxx-xxx.ap-southeast-2.compute.internal%7Cip-xx-xxx-xxx-xxx.ap-southeast-2.compute.internal%7Cip-xx-xxx-xxx-xxx.ap-southeast-2.co mpute.internal%22%7D%0A++%29%0A%29%0Aor+sum+by+%28node%29+%28%0A++1+-+irate%28%0A++++windows_cpu_time_total%7Bmode%3D%22idle%22%2C+job%3D%22windows-exporter%22%2Cnode%3D~%22ip-xx-xxx-xxx-xxx.ap-southeast-2.compute.internal%7Cip-xx-xxx-xxx-xxx.ap-southeast-2.compute.internal%7Cip-xx-xxx-xxx-xxx.ap-southeast-2.compute.internal%7Cip-xx-xxx-xxx-xxx.ap-southeast-2.compute.internal%7Cip-xx-xxx-xxx-xxx.ap-southeast-2.compute.internal%7Cip-xx-xxx-xxx-xxx.ap-southeast-2.compute.internal%22%7D%5B4m%5D%0A++%29%0A%29%0A&time=1707971937.171 200 OK I0215 04:38:57.174974 1 api.go:107] Response Body: {"status":"success","data":{"resultType":"vector","result":[]}} I0215 04:38:57.178115 1 api.go:88] GET http://prometheus-k8s.monitoring-system.svc:9090/api/v1/query?query=sum+by+%28instance%29+%28%0A++node_memory_MemTotal_bytes%7Bjob%3D%22node-exporter%22%2Cinstance%3D~%22ip-xx-xxx-xxx-xxx.ap-southeast-2.compute.internal%7Cip-xx-xxx-xxx-xxx.ap-southeast-2.compute.interna l%7Cip-xx-xxx-xxx-xxx.ap-southeast-2.compute.internal%7Cip-xx-xxx-xxx-xxx.ap-southeast-2.compute.internal%7Cip-xx-xxx-xxx-xxx.ap-southeast-2.compute.internal%7Cip-xx-xxx-xxx-xxx.ap-southeast-2.compute.internal%22%7D%0A++-%0A++node_memory_MemAvailable_bytes%7Bjob%3D%22node-exporter%22%2Cinstance%3D~%22ip-10-161-218- 141.ap-southeast-2.compute.internal%7Cip-xx-xxx-xxx-xxx.ap-southeast-2.compute.internal%7Cip-xx-xxx-xxx-xxx.ap-southeast-2.compute.internal%7Cip-xx-xxx-xxx-xxx.ap-southeast-2.compute.internal%7Cip-xx-xxx-xxx-xxx.ap-southeast-2.compute.internal%7Cip-xx-xxx-xxx-xxx.ap-southeast-2.compute.internal%22%7D%0A%29%0Aor+sum+ by+%28instance%29+%28%0A++windows_cs_physical_memory_bytes%7Bjob%3D%22windows-exporter%22%2Cinstance%3D~%22ip-xx-xxx-xxx-xxx.ap-southeast-2.compute.internal%7Cip-xx-xxx-xxx-xxx.ap-southeast-2.compute.internal%7Cip-xx-xxx-xxx-xxx.ap-southeast-2.compute.internal%7Cip-xx-xxx-xxx-xxx.ap-southeast-2.compute.internal%7Ci p-10-161-216-168.ap-southeast-2.compute.internal%7Cip-xx-xxx-xxx-xxx.ap-southeast-2.compute.internal%22%7D%0A++-%0A++windows_memory_available_bytes%7Bjob%3D%22windows-exporter%22%2Cinstance%3D~%22ip-xx-xxx-xxx-xxx.ap-southeast-2.compute.internal%7Cip-xx-xxx-xxx-xxx.ap-southeast-2.compute.internal%7Cip-xx-xxx-xxx-xxx .ap-southeast-2.compute.internal%7Cip-xx-xxx-xxx-xxx.ap-southeast-2.compute.internal%7Cip-xx-xxx-xxx-xxx.ap-southeast-2.compute.internal%7Cip-xx-xxx-xxx-xxx.ap-southeast-2.compute.internal%22%7D%0A%29%0A&time=1707971937.171 200 OK I0215 04:38:57.178231 1 api.go:107] Response Body: {"status":"success","data":{"resultType":"vector","result":[{"metric":{"instance":"ip-xx-xxx-xxx-xxx.ap-southeast-2.compute.internal"},"value":[1707971937.171,"1563303936"]},{"metric":{"instance":"ip-xx-xxx-xxx-xxx.ap-southeast-2.compute.internal"},"value": [1707971937.171,"3037470720"]},{"metric":{"instance":"ip-xx-xxx-xxx-xxx.ap-southeast-2.compute.internal"},"value":[1707971937.171,"2406985728"]},{"metric":{"instance":"ip-xx-xxx-xxx-xxx.ap-southeast-2.compute.internal"},"value":[1707971937.171,"1566240768"]},{"metric":{"instance":"ip-xx-xxx-xxx-xxx.ap-southeast-2.c ompute.internal"},"value":[1707971937.171,"3543482368"]},{"metric":{"instance":"ip-xx-xxx-xxx-xxx.ap-southeast-2.compute.internal"},"value":[1707971937.171,"1575890944"]}]}} I0215 04:38:57.178655 1 provider.go:291] missing CPU for node "ip-xx-xxx-xxx-xxx.ap-southeast-2.compute.internal", skipping I0215 04:38:57.178670 1 provider.go:291] missing CPU for node "ip-xx-xxx-xxx-xxx.ap-southeast-2.compute.internal", skipping I0215 04:38:57.178676 1 provider.go:291] missing CPU for node "ip-xx-xxx-xxx-xxx.ap-southeast-2.compute.internal", skipping I0215 04:38:57.178682 1 provider.go:291] missing CPU for node "ip-xx-xxx-xxx-xxx.ap-southeast-2.compute.internal", skipping I0215 04:38:57.178688 1 provider.go:291] missing CPU for node "ip-xx-xxx-xxx-xxx.ap-southeast-2.compute.internal", skipping I0215 04:38:57.178694 1 provider.go:291] missing CPU for node "ip-xx-xxx-xxx-xxx.ap-southeast-2.compute.internal", skipping I0215 04:38:57.178918 1 httplog.go:132] "HTTP" verb="LIST" URI="/apis/metrics.k8s.io/v1beta1/nodes" latency="8.096979ms" userAgent="kubectl/v1.26.7 (linux/amd64) kubernetes/89a3d86" audit-ID="4d2e7d1c-5107-4e3a-8caa-67c0c4e2ff6f" srcIP="10.161.221.68:53746" resp=200 I0215 04:38:57.221383 1 handler.go:153] prometheus-metrics-adapter: GET "/livez" satisfied by nonGoRestful I0215 04:38:57.221410 1 pathrecorder.go:241] prometheus-metrics-adapter: "/livez" satisfied by exact match I0215 04:38:57.221511 1 handler.go:153] prometheus-metrics-adapter: GET "/readyz" satisfied by nonGoRestful I0215 04:38:57.221539 1 pathrecorder.go:241] prometheus-metrics-adapter: "/readyz" satisfied by exact match I0215 04:38:57.221513 1 httplog.go:132] "HTTP" verb="GET" URI="/livez" latency="238.051µs" userAgent="kube-probe/1.26+" audit-ID="ac3d0baa-550b-4ea1-a156-563e84503bf4" srcIP="10.161.218.141:24110" resp=200 I0215 04:38:57.221592 1 shared_informer.go:341] caches populated I0215 04:38:57.221668 1 httplog.go:132] "HTTP" verb="GET" URI="/readyz" latency="255.747µs" userAgent="kube-probe/1.26+" audit-ID="0813d31c-df53-4066-b23d-0edea4e8db04" srcIP="10.161.218.141:24112" resp=200 ```Anything else we need to know?:
Environment:
kubectl version): 1.26