search

google / cadvisor

Analyzes resource usage and performance characteristics of running containers.
Other
17.16k stars 2.32k forks source link

processStatsFromProcs High CPU Usage with High Number of File Descriptors #3233

Open Champ-Goblem opened 1 year ago

Champ-Goblem commented 1 year ago

On our nodes with a large number of pods (in excess of 150 in most cases) which may exhibit a large number of open file descriptors, we see that the kubelet running on these nodes is often using anywhere between 0.5-2 threads out of 24.

When flame-graphing the kubelet process we can see that the majority of the time is spent in the processStatsFromProcs function:

kubelet-flame-graph

This is due to the fact that the function performs a system call per /proc/<pid>/fd directory (readdir) and a system call per file descriptor listed in this directory (readlink). If there are a large number of open file descriptors and a large number of process running on these nodes, the overheads of these system calls add a lot of impact to the process.

Is there a way to optimise this function to reduce the overall CPU usage of the kubelet?

sibo-git commented 1 year ago

没有人处理吗？

iwankgb commented 1 year ago

From the top of my had I can't see how to fix it (in terms of replacing readlink() with another syscall. You can disable process stats collections by appending process to list of disable metric kinds, e.g. cadvisor --disable_metrics=process. It may be possible to pass the same argument to kubelet.

sibo-git commented 1 year ago

How to repair it? my kubernetes version is 1.23.0

iwankgb commented 1 year ago

After looking into the problem I, sadly, have to state that io_uring does not support readlink syscall. It does not seem to be possible to get same information using other, supported, syscalls.

DronNick commented 1 month ago

On our nodes with a large number of pods (in excess of 150 in most cases) which may exhibit a large number of open file descriptors, we see that the kubelet running on these nodes is often using anywhere between 0.5-2 threads out of 24.

There must be millions of file descriptors on your pods pids, if it takes a complete CPU core.

Champ-Goblem commented 1 month ago

Yes there will be, it is one of the downsides of running Kata with virtio-fs, it has a internal inode store that it uses which caches open file descriptors so it does not have to keep reopening them for each request.