Open jibinrajck opened 5 months ago
What happened: kubectl top pods returning error: Metrics not available for pod default/awscli, age: 177h19m9.789235105s
& k top nodes returns
error: metrics not available yet
What you expected to happen: Should retrun valid responses
Anything else we need to know?:
Logs :
[ec2-user@10.161.133.102 2024-02-05_09:59:16 UTC ~]$ k top node --v=8 I0205 10:00:43.902192 21565 loader.go:373] Config loaded from file: /home/ec2-user/.kube/config I0205 10:00:43.903956 21565 round_trippers.go:463] GET https://xxxxxxxxxxxxxxxxxxxxx.gr7.ap-southeast-2.eks.amazonaws.com/api I0205 10:00:43.903996 21565 round_trippers.go:469] Request Headers: I0205 10:00:43.904014 21565 round_trippers.go:473] Accept: application/json;g=apidiscovery.k8s.io;v=v2beta1;as=APIGroupDiscoveryList,application/json I0205 10:00:43.904030 21565 round_trippers.go:473] User-Agent: kubectl/v1.26.7 (linux/amd64) kubernetes/89a3d86 I0205 10:00:44.054485 21565 round_trippers.go:574] Response Status: 200 OK in 150 milliseconds I0205 10:00:44.057617 21565 round_trippers.go:577] Response Headers: I0205 10:00:44.057727 21565 round_trippers.go:580] X-Kubernetes-Pf-Flowschema-Uid: 70464fe9-5fd3-4a02-abad-d4bdf1640edb I0205 10:00:44.057747 21565 round_trippers.go:580] X-Kubernetes-Pf-Prioritylevel-Uid: 934dbb0f-5e8b-4602-baaa-b960847fbfb2 I0205 10:00:44.057763 21565 round_trippers.go:580] Content-Length: 170 I0205 10:00:44.057780 21565 round_trippers.go:580] Date: Mon, 05 Feb 2024 10:00:44 GMT I0205 10:00:44.057794 21565 round_trippers.go:580] Audit-Id: de54b295-eb5c-4091-933f-65ad60069091 I0205 10:00:44.057806 21565 round_trippers.go:580] Cache-Control: no-cache, private I0205 10:00:44.057821 21565 round_trippers.go:580] Content-Type: application/json I0205 10:00:44.057857 21565 request.go:1171] Response Body: {"kind":"APIVersions","versions":["v1"],"serverAddressByClientCIDRs":[{"clientCIDR":"0.0.0.0/0","serverAddress":"ip-123-45-678-98.ap-southeast-2.compute.internal:443"}]} I0205 10:00:44.058534 21565 round_trippers.go:463] GET https://xxxxxxxxxxxxxxxxxxxxx.gr7.ap-southeast-2.eks.amazonaws.com/apis I0205 10:00:44.058629 21565 round_trippers.go:469] Request Headers: I0205 10:00:44.058650 21565 round_trippers.go:473] Accept: application/json;g=apidiscovery.k8s.io;v=v2beta1;as=APIGroupDiscoveryList,application/json I0205 10:00:44.058667 21565 round_trippers.go:473] User-Agent: kubectl/v1.26.7 (linux/amd64) kubernetes/89a3d86 I0205 10:00:44.059521 21565 round_trippers.go:574] Response Status: 200 OK in 0 milliseconds I0205 10:00:44.059546 21565 round_trippers.go:577] Response Headers: I0205 10:00:44.059563 21565 round_trippers.go:580] Audit-Id: 8c14f103-4ffd-42e1-b60f-8d34d871cbcd I0205 10:00:44.059588 21565 round_trippers.go:580] Cache-Control: no-cache, private I0205 10:00:44.059604 21565 round_trippers.go:580] Content-Type: application/json I0205 10:00:44.059619 21565 round_trippers.go:580] X-Kubernetes-Pf-Flowschema-Uid: 70464fe9-5fd3-4a02-abad-d4bdf1640edb I0205 10:00:44.059636 21565 round_trippers.go:580] X-Kubernetes-Pf-Prioritylevel-Uid: 934dbb0f-5e8b-4602-baaa-b960847fbfb2 I0205 10:00:44.059653 21565 round_trippers.go:580] Date: Mon, 05 Feb 2024 10:00:44 GMT I0205 10:00:44.060253 21565 request.go:1171] Response Body: {"kind":"APIGroupList","apiVersion":"v1","groups":[{"name":"apiregistration.k8s.io","versions":[{"groupVersion":"apiregistration.k8s.io/v1","version":"v1"}],"preferredVersion":{"groupVersion":"apiregistration.k8s.io/v1","version":"v1"}},{"name":"apps","versions":[{"groupVersion":"apps/v1","version":"v1"}],"preferredVersion":{"groupVersion":"apps/v1","version":"v1"}},{"name":"events.k8s.io","versions":[{"groupVersion":"events.k8s.io/v1","version":"v1"}],"preferredVersion":{"groupVersion":"events.k8s.io/v1","version":"v1"}},{"name":"authentication.k8s.io","versions":[{"groupVersion":"authentication.k8s.io/v1","version":"v1"}],"preferredVersion":{"groupVersion":"authentication.k8s.io/v1","version":"v1"}},{"name":"authorization.k8s.io","versions":[{"groupVersion":"authorization.k8s.io/v1","version":"v1"}],"preferredVersion":{"groupVersion":"authorization.k8s.io/v1","version":"v1"}},{"name":"autoscaling","versions":[{"groupVersion":"autoscaling/v2","version":"v2"},{"groupVersion":"autoscaling/v1","version":"v1"}], [truncated 4396 chars] I0205 10:00:44.062517 21565 round_trippers.go:463] GET https://xxxxxxxxxxxxxxxxxxxxx.gr7.ap-southeast-2.eks.amazonaws.com/apis/metrics.k8s.io/v1beta1/nodes I0205 10:00:44.062611 21565 round_trippers.go:469] Request Headers: I0205 10:00:44.062632 21565 round_trippers.go:473] Accept: application/vnd.kubernetes.protobuf, */* I0205 10:00:44.062651 21565 round_trippers.go:473] User-Agent: kubectl/v1.26.7 (linux/amd64) kubernetes/89a3d86 I0205 10:00:44.072123 21565 round_trippers.go:574] Response Status: 200 OK in 9 milliseconds I0205 10:00:44.073105 21565 round_trippers.go:577] Response Headers: I0205 10:00:44.073131 21565 round_trippers.go:580] X-Kubernetes-Pf-Prioritylevel-Uid: 934dbb0f-5e8b-4602-baaa-b960847fbfb2 I0205 10:00:44.073147 21565 round_trippers.go:580] Content-Length: 61 I0205 10:00:44.073162 21565 round_trippers.go:580] Audit-Id: ee0f3e0e-824f-4368-83ff-d888f818f2ef I0205 10:00:44.073175 21565 round_trippers.go:580] Cache-Control: no-cache, private I0205 10:00:44.073188 21565 round_trippers.go:580] Cache-Control: no-cache, private I0205 10:00:44.073202 21565 round_trippers.go:580] Content-Type: application/vnd.kubernetes.protobuf I0205 10:00:44.073219 21565 round_trippers.go:580] Date: Mon, 05 Feb 2024 10:00:44 GMT I0205 10:00:44.073234 21565 round_trippers.go:580] X-Kubernetes-Pf-Flowschema-Uid: 70464fe9-5fd3-4a02-abad-d4bdf1640edb I0205 10:00:44.073272 21565 request.go:1169] Response Body: 00000000 6b 38 73 00 0a 29 0a 16 6d 65 74 72 69 63 73 2e |k8s..)..metrics.| 00000010 6b 38 73 2e 69 6f 2f 76 31 62 65 74 61 31 12 0f |k8s.io/v1beta1..| 00000020 4e 6f 64 65 4d 65 74 72 69 63 73 4c 69 73 74 12 |NodeMetricsList.| 00000030 08 0a 06 0a 00 12 00 1a 00 1a 00 22 00 |...........".| error: metrics not available yet
[ec2-user@10.161.133.102 2024-02-05_10:11:27 UTC ~]$ kn logs metrics-server-5c5f997c95-gkt9t I0205 09:57:11.782601 1 serving.go:374] Generated self-signed cert (/tmp/apiserver.crt, /tmp/apiserver.key) I0205 09:57:12.211023 1 handler.go:275] Adding GroupVersion metrics.k8s.io v1beta1 to ResourceManager I0205 09:57:12.320721 1 requestheader_controller.go:169] Starting RequestHeaderAuthRequestController I0205 09:57:12.320934 1 shared_informer.go:311] Waiting for caches to sync for RequestHeaderAuthRequestController I0205 09:57:12.320967 1 configmap_cafile_content.go:202] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file" I0205 09:57:12.321004 1 shared_informer.go:311] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file I0205 09:57:12.321104 1 configmap_cafile_content.go:202] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::client-ca-file" I0205 09:57:12.321116 1 shared_informer.go:311] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file I0205 09:57:12.322559 1 secure_serving.go:213] Serving securely on [::]:10250 I0205 09:57:12.322618 1 dynamic_serving_content.go:132] "Starting controller" name="serving-cert::/tmp/apiserver.crt::/tmp/apiserver.key" I0205 09:57:12.322784 1 tlsconfig.go:240] "Starting DynamicServingCertificateController" I0205 09:57:12.421066 1 shared_informer.go:318] Caches are synced for RequestHeaderAuthRequestController I0205 09:57:12.421150 1 shared_informer.go:318] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file I0205 09:57:12.421198 1 shared_informer.go:318] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-fil
Environment:
Kubernetes distribution (GKE, EKS, Kubeadm, the hard way, etc.): EKS
Container Network Setup (flannel, calico, etc.): AWS VPC CNI
Kubernetes version (use kubectl version): 1.26
kubectl version
Metrics Server manifest kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml
/kind bug
/kind support /triage accepted /assign @yangjunmyfm192085
hi, @jibinrajck It seems that metrics server does not obtain the metrics information of pods or nodes. Can you look at the logs of metrics server?
What happened: kubectl top pods returning error: Metrics not available for pod default/awscli, age: 177h19m9.789235105s
& k top nodes returns
error: metrics not available yet
What you expected to happen: Should retrun valid responses
Anything else we need to know?:
Logs :
Environment:
Kubernetes distribution (GKE, EKS, Kubeadm, the hard way, etc.): EKS
Container Network Setup (flannel, calico, etc.): AWS VPC CNI
Kubernetes version (use
kubectl version
): 1.26Metrics Server manifest kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml
spoiler for Metrics Server manifest:
spoiler for Kubelet config:
spoiler for Metrics Server logs:
spolier for Status of Metrics API:
```sh kubectl describe apiservice v1beta1.metrics.k8s.io ```/kind bug