google / cadvisor

Analyzes resource usage and performance characteristics of running containers.
Other
17.14k stars 2.32k forks source link

Failed to update stats for container .. error binary.Read: invalid type int32 #3574

Open thunderbird86 opened 3 months ago

thunderbird86 commented 3 months ago

Hello, I've run cAdvisor on EKS cluster, and found that half on metrics not present, and it depends on instances.

I'm getting following error,

W0807 10:23:50.887737 3249444 container.go:588] Failed to update stats for container "/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73387ff3_7d68_4b26_ace5_ab12e2c91a7f.slice/cri-containerd-1948d1e50a9b535ce6babd400f919f47dd20bd36014a8b24ae0d779b9ceec6b1.scope": failed to get load stat for "/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73387ff3_7d68_4b26_ace5_ab12e2c91a7f.slice/cri-containerd-1948d1e50a9b535ce6babd400f919f47dd20bd36014a8b24ae0d779b9ceec6b1.scope" - path "/sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73387ff3_7d68_4b26_ace5_ab12e2c91a7f.slice/cri-containerd-1948d1e50a9b535ce6babd400f919f47dd20bd36014a8b24ae0d779b9ceec6b1.scope", error binary.Read: invalid type int32

while on other instances it works fine.

I found thats main difference between working/notworking instances is kernel version

 Fails:
AMI name: amazon-eks-node-al2023-x86_64-standard-1.30-v20240703
KV: 6.1.94-99.176.amzn2023.x86_64
CR: containerd://1.7.11
Kubelet: v1.30.0-eks-036c24b

Working: 
AMI Name: amazon-eks-node-1.30-v20240729
KV: 5.10.220-209.869.amzn2.x86_64
CR: containerd://1.7.11
Kubelet: v1.30.2-eks-1552ad0

Version of cAdvisor: v0.49.1 and v0.50.0

chengjoey commented 2 months ago

3430

https://github.com/google/cadvisor/blob/c15f44e578c77800b1b82a7bbb67614364f4aedc/utils/cpuload/netlink/netlink.go#L207-L214