container_network* Metrics Missing Labels after K8S 1.30.1 Upgrade

[johnmwood]

My team recently upgraded from K8S v1.29.5 to v1.30.1. We noticed all metrics for container_network* immediately drop labels. When checking our metrics endpoint, here are a few examples:

kubectl get --raw /api/v1/nodes/NODE/proxy/metrics/cadvisor 

container_network_receive_errors_total{container="",id="/",image="",interface="bond0",name="",namespace="",pod=""} 1634 1718830359399

container_network_receive_errors_total{container="",id="/",image="",interface="bond0",name="",namespace="",pod=""} 1634 1718830359399

container_network_receive_packets_dropped_total{container="",id="/",image="",interface="bond0",name="",namespace="",pod=""} 2.907596e+06 1718830359399

This has broken our existing prometheus queries for these metrics.

Environment

• Runtime: cri-o
• Image: Debian Bookworm
• K8S: v1.30.1
• cgroup-driver: systemd

Potentially related issues

• K8s/cri-o broken for cgroup-driver != systemd
• Not able to view container names for network metrics
• Cadvisor Metrics Label contain empty Strings as values (container="", image="",...)

[nicolas-laduguie]

Hi, any news on that issue ?

[nick-oconnor]

@haircommander any ideas what could have caused this? I lost all container_network_ metrics on 1.30 w/ cri-o + systemd. This seems like another cri-o/systemd/cadvisor issue.