Metric server is failing to get Memory Usage

[mohammadkhavari]

What happened: Metric server is failing to report pods metrics:

kubectl top pod
error: Metrics not available for pod default/consul-f99d7d6ff-5ghxk, age: 269h15m31.270817426s

however it works for nodes:

kubectl top node NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% khavari 543m 9% 9147Mi 42%

What you expected to happen:

Anything else we need to know?: I have deplyed metric server on v=4 and from logs it can be inferred that its failing to fetch memory usage for pods: -> MemoryUsage:0 I0614 06:37:14.529323 1 decode.go:189] "Failed getting complete container metric" containerName="hotel-reserv-rate-mmc" containerMetric={StartTime:2022-06-14 06:36:04 +0000 UTC Timestamp:2022-06-14 06:37:14.492 +0000 UTC CumulativeCpuUsed:30673360 MemoryUsage:0} I0614 06:37:14.529335 1 decode.go:97] "Failed getting complete Pod metric" pod="default/memcached-rate-7ddf457845-xr8tv"

I have installed metric-server by: kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml

and then change args by kubectl edit: add hostNetwork: true and change args to:

Environment:

• Kubernetes distribution (GKE, EKS, Kubeadm, the hard way, etc.): kubeadm

• Container Network Setup (flannel, calico, etc.): flannel

• Kubernetes version (use kubectl version): v1.24.1

• Metrics Server manifest

spoiler for Metrics Server manifest:

• Kubelet config:

spoiler for Kubelet config:

• Metrics server logs:

spoiler for Metrics Server logs:

[metric-server-log.txt](https://github.com/kubernetes-sigs/metrics-server/files/8897620/metric-server-log.txt)

• Status of Metrics API:

  spolier for Status of Metrics API:

  ``` kubectl describe apiservice v1beta1.metrics.k8s.io ``` ` kubectl describe apiservice v1beta1.metrics.k8s.io ubuntu@khavari f:spec: f:group: f:groupPriorityMinimum: f:insecureSkipTLSVerify: f:service: .: f:name: f:namespace: f:port: f:version: f:versionPriority: Manager: kubectl-client-side-apply Operation: Update Time: 2022-06-14T03:44:00Z Resource Version: 1701370 UID: 56c7a2fd-0c90-4209-b749-d8f3b7eb4e23 Spec: Group: metrics.k8s.io Group Priority Minimum: 100 Insecure Skip TLS Verify: true Service: Name: metrics-server Namespace: kube-system Port: 443 Version: v1beta1 Version Priority: 100 Status: Conditions: Last Transition Time: 2022-06-14T06:33:21Z Message: all checks passed Reason: Passed Status: True Type: Available Events: `

/kind bug

[serathius]

Please read https://github.com/kubernetes-sigs/metrics-server/blob/master/KNOWN_ISSUES.md#kubelet-doesnt-report-pod-metrics

[mohammadkhavari]

@serathius Thanks for your consideration I have fetched metrics to figure out whether the problem originates from kubelet or not. by checking summary API I have found that kubelet is reporting pods memory metrics. but metric api is not. It is not reporting memory usage.

kubectl get --raw /api/v1/nodes/$NODE_NAME/proxy/stats/summary > stst-summary.log

but metric server is not reporting it. kubectl logs -n kube-system metrics-server-7478666b7c-zxnsc

I0614 06:53:15.368682 1 decode.go:189] "Failed getting complete container metric" containerName="hotel-reserv-geo" containerMetric={StartTime:2022-06-14 06:38:42 +0000 UTC Timestamp:2022-06-14 06:53:15.251 +0000 UTC CumulativeCpuUsed:358344047 MemoryUsage:0} I0614 06:53:15.368706 1 decode.go:97] "Failed getting complete Pod metric" pod="default/geo-bdb894c47-vwlj2"

complete json response: stst-summary.log

[yangjunmyfm192085]

If the version of metrics-server is 0.6.x, I think you should use the command

kubectl get --raw /api/v1/nodes/$NODE_NAME/proxy/metrics/resource

[mohammadkhavari]

This Issue is not from metric-server's side. both /metrics/resources and stats/summary are logging zero container memory_working_set_bytes however pod's memory_working_set_bytes is not zero.

container_memory_working_set_bytes{container="etcd",namespace="kube-system",pod="etcd-khavari"} 0 1655320691867 ...skip... pod_memory_working_set_bytes{namespace="kube-system",pod="etcd-khavari"} 1.07737088e+08 1655320677346 I'm closing this issue.