google / cadvisor

Analyzes resource usage and performance characteristics of running containers.
Other
17.01k stars 2.31k forks source link

container_network* Metrics Missing Labels after K8S 1.30.1 Upgrade #3551

Open johnmwood opened 3 months ago

johnmwood commented 3 months ago

My team recently upgraded from K8S v1.29.5 to v1.30.1. We noticed all metrics for container_network* immediately drop labels. When checking our metrics endpoint, here are a few examples:

kubectl get --raw /api/v1/nodes/NODE/proxy/metrics/cadvisor 

container_network_receive_errors_total{container="",id="/",image="",interface="bond0",name="",namespace="",pod=""} 1634 1718830359399

container_network_receive_errors_total{container="",id="/",image="",interface="bond0",name="",namespace="",pod=""} 1634 1718830359399

container_network_receive_packets_dropped_total{container="",id="/",image="",interface="bond0",name="",namespace="",pod=""} 2.907596e+06 1718830359399

This has broken our existing prometheus queries for these metrics.

Environment

Potentially related issues