Open cyrus-mc opened 2 years ago
@ysksuzuki pinging you here to see if this is related to #3026 that was recently closed, and I think the fix is awaiting release.
Is there any update on this? We have the same problem running cadvisor v0.46.0
on our cluster. Changing back to cgroup v1
restores also other metrics like container_memory_max_usage_bytes
.
Is there any update? We are also missing these metrics when using cgroup v2.
Spent some time looking into the code, it seems the underlying crun library doesn't populate MemoryStats.Usage.MaxUsage
when Cgroups V2 is used. So the code at https://github.com/google/cadvisor/blob/fdd3d9182bea6f7f11e4f934631c4abef3aa0584/container/libcontainer/handler.go#L801 is insufficient.
A possible solution would be to place ret.Memory.MaxUsage = s.MemoryStats.Stats["peak"]
in the if cgroups.IsCgroup2UnifiedMode()
block. Don't have a V2 system handy right now to test with :(
Went into the rabbit hole a bit further. It seems peak
is only available since kernel 5.19 (specifically commit https://github.com/torvalds/linux/commit/8e20d4b332660a32e842e20c34cfc3b3456bc6dc).
So, next would be to test on 6.1 kernel.
I'm beginning to fix this in https://github.com/opencontainers/runc/pull/4038
I'm beginning to fix this in opencontainers/runc#4038
I think this fix made it's way into runc 1.11.0. Since then, I've seen that COS-113 mentions runc 1.12.0. And so does cAdvisor 0.49.0.
But I guess the fix suggested here would still need to be implemented before the metric would be exposed by cAdvisor. Would anyone know the status of this?
This seems to be fixed. I've seen both container_cpu_cfs_throttled_*
and container_memory_max_usage_bytes
being present and not zero when run on a node with kubelet v1.30 (has cadvisor 0.49.0) when cgroup v2 is used.
Went into the rabbit hole a bit further. It seems
peak
is only available since kernel 5.19 (specifically commit torvalds/linux@8e20d4b). So, next would be to test on 6.1 kernel.
@mindw Thanks for digging. I stumbled across your comments while trying to measure the peak memory use of cert-manager components.
I was testing on Kind on a Windows WSL2 virtual machine and observed container_memory_max_usage_bytes
having only zero values.
The default WSL2 kernel is v5.15 but happily a new WSL2 v6 kernel is soon to be released so I'll report back when that is available.
This might be related to https://github.com/google/cadvisor/issues/3026 (which I am not sure has been released yet).
On nodes running
cgroup
v1 the following metrics such ascontainer_cpu_cfs_throttled_*
are returned. Ifcadvisor
is run on nodes withcgroup
v2 enabled those metrics are not returned.There could be others but these are the ones I noted when attempting to troubleshoot an issue. To verify I ran the latest of the 0.39.x release and 0.43.x and both exhibited the same behavior.