docker / for-linux

Docker Engine for Linux
https://docs.docker.com/engine/installation/
755 stars 86 forks source link

No CPU usage on docker stats #1477

Closed kdryetyln closed 7 months ago

kdryetyln commented 12 months ago

Hello, I am using Docker rootless on Rocky 8. Cgroup is v2. We switched to V2 because we could not define a limitation for containers. However, after that, we started not being able to see and collect CPU usage. We cannot understand exactly what causes the problem. I am adding here all the information I think will be useful. I hope we can reach a solution on this issue. Thank you in advance for your opinions.

[user@xyz ~]$ cat /etc/os-release
NAME=“Rocky Linux”
VERSION=“8.8 (Green Obsidian)”
ID=“rocky”
ID_LIKE=“rhel centos fedora”
VERSION_ID=“8.8”
PLATFORM_ID=“platform:el8”
PRETTY_NAME=“Rocky Linux 8.8 (Green Obsidian)”
ANSI_COLOR=“0;32”
LOGO=“fedora-logo-icon”
CPE_NAME=“cpe:/o:rocky:rocky:8:GA”
HOME_URL=“https://rockylinux.org/”
BUG_REPORT_URL=“https://bugs.rockylinux.org/”
SUPPORT_END=“2029-05-31”
ROCKY_SUPPORT_PRODUCT=“Rocky-Linux-8”
ROCKY_SUPPORT_PRODUCT_VERSION=“8.8”
REDHAT_SUPPORT_PRODUCT=“Rocky Linux”
REDHAT_SUPPORT_PRODUCT_VERSION=“8.8”
[user@xyz ~]$ uname -a
Linux xyz 4.18.0-477.27.1.el8_8.x86_64 #1 SMP Wed Sep 20 15:55:39 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
[user@xyz ~]$ uname -r
4.18.0-477.27.1.el8_8.x86_64
[user@xyz ~]$ lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 10
On-line CPU(s) list: 0-9
Thread(s) per core: 1
Core(s) per socket: 1
Socket(s): 10
NUMA node(s): 1
Vendor ID: GenuineIntel
CPU family: 6
Model: 106
Model name: Intel(R) Xeon(R) Gold 6342 CPU @ 2.80GHz
Stepping: 6
CPU MHz: 2793.437
BogoMIPS: 5586.87
Hypervisor vendor: VMware
Virtualization type: full
L1d cache: 48K
L1i cache: 32K
L2 cache: 1280K
L3 cache: 36864K
NUMA node0 CPU(s): 0-9
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon rep_good nopl xtopology tsc_reliable nonstop_tsc cpuid pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single ssbd ibrs ibpb stibp ibrs_enhanced fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves wbnoinvd arat avx512vbmi umip pku ospke avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg avx512_vpopcntdq rdpid fsrm md_clear flush_l1d arch_capabilities
[user@xyz ~]$ docker info
Client:
Version: 24.0.7
Context: default
Debug Mode: false

Server:
Containers: 2
Running: 0
Paused: 0
Stopped: 2
Images: 3
Server Version: 24.0.7
Storage Driver: fuse-overlayfs
Logging Driver: json-file
Cgroup Driver: systemd
Cgroup Version: 2
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc io.containerd.runc.v2
Default Runtime: runc
Init Binary: docker-init
containerd version: abc
runc version: v1.1.9-0-gccaecfc
init version: de40ad0
Security Options:
seccomp
Profile: builtin
rootless
cgroupns
Kernel Version: 4.18.0-477.27.1.el8_8.x86_64
Operating System: Rocky Linux 8.8 (Green Obsidian)
OSType: linux
Architecture: x86_64
CPUs: 10
Total Memory: 51.3GiB
Name: xyz
ID: dd5e0e85-1f30-4103-b6d7-2f0d20ed4c50
Docker Root Dir: docker-paths
Debug Mode: false
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
Product License: Community Engine

WARNING: No cpuset support
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled
[user@xyz ~]$

./check-config.sh
warning: /proc/config.gz does not exist, searching other paths for kernel config …
info: reading kernel config from /boot/config-4.18.0-477.27.1.el8_8.x86_64 …

Generally Necessary:

cgroup hierarchy: cgroupv2
Controllers:
cpu: available
cpuset: available
io: available
memory: available
pids: available
CONFIG_NAMESPACES: enabled
CONFIG_NET_NS: enabled
CONFIG_PID_NS: enabled
CONFIG_IPC_NS: enabled
CONFIG_UTS_NS: enabled
CONFIG_CGROUPS: enabled
CONFIG_CGROUP_CPUACCT: enabled
CONFIG_CGROUP_DEVICE: enabled
CONFIG_CGROUP_FREEZER: enabled
CONFIG_CGROUP_SCHED: enabled
CONFIG_CPUSETS: enabled
CONFIG_MEMCG: enabled
CONFIG_KEYS: enabled
CONFIG_VETH: enabled (as module)
CONFIG_BRIDGE: enabled (as module)
CONFIG_BRIDGE_NETFILTER: enabled (as module)
CONFIG_IP_NF_FILTER: enabled (as module)
CONFIG_IP_NF_MANGLE: enabled (as module)
CONFIG_IP_NF_TARGET_MASQUERADE: enabled (as module)
CONFIG_NETFILTER_XT_MATCH_ADDRTYPE: enabled (as module)
CONFIG_NETFILTER_XT_MATCH_CONNTRACK: enabled (as module)
CONFIG_NETFILTER_XT_MATCH_IPVS: enabled (as module)
CONFIG_NETFILTER_XT_MARK: enabled (as module)
CONFIG_IP_NF_NAT: enabled (as module)
CONFIG_NF_NAT: enabled (as module)
CONFIG_POSIX_MQUEUE: enabled
CONFIG_NF_NAT_IPV4: missing
CONFIG_NF_NAT_NEEDED: enabled
CONFIG_CGROUP_BPF: enabled
Optional Features:

CONFIG_USER_NS: enabled
CONFIG_SECCOMP: enabled
CONFIG_SECCOMP_FILTER: enabled
CONFIG_CGROUP_PIDS: enabled
CONFIG_MEMCG_SWAP: enabled
CONFIG_MEMCG_SWAP_ENABLED: missing
(cgroup swap accounting is currently not enabled, you can enable it by setting boot option “swapaccount=1”)
CONFIG_LEGACY_VSYSCALL_EMULATE: enabled
CONFIG_IOSCHED_CFQ: missing
CONFIG_CFQ_GROUP_IOSCHED: missing
CONFIG_BLK_CGROUP: enabled
CONFIG_BLK_DEV_THROTTLING: enabled
CONFIG_CGROUP_PERF: enabled
CONFIG_CGROUP_HUGETLB: enabled
CONFIG_NET_CLS_CGROUP: enabled
CONFIG_CGROUP_NET_PRIO: enabled
CONFIG_CFS_BANDWIDTH: enabled
CONFIG_FAIR_GROUP_SCHED: enabled
CONFIG_IP_NF_TARGET_REDIRECT: enabled (as module)
CONFIG_IP_VS: enabled (as module)
CONFIG_IP_VS_NFCT: enabled
CONFIG_IP_VS_PROTO_TCP: enabled
CONFIG_IP_VS_PROTO_UDP: enabled
CONFIG_IP_VS_RR: enabled (as module)
CONFIG_SECURITY_SELINUX: enabled
CONFIG_SECURITY_APPARMOR: missing
CONFIG_EXT4_FS: enabled (as module)
CONFIG_EXT4_FS_POSIX_ACL: enabled
CONFIG_EXT4_FS_SECURITY: enabled
Network Drivers:
“overlay”:
CONFIG_VXLAN: enabled (as module)
CONFIG_BRIDGE_VLAN_FILTERING: enabled
Optional (for encrypted networks):
CONFIG_CRYPTO: enabled
CONFIG_CRYPTO_AEAD: enabled
CONFIG_CRYPTO_GCM: enabled
CONFIG_CRYPTO_SEQIV: enabled (as module)
CONFIG_CRYPTO_GHASH: enabled
CONFIG_XFRM: enabled
CONFIG_XFRM_USER: enabled
CONFIG_XFRM_ALGO: enabled
CONFIG_INET_ESP: enabled (as module)
CONFIG_NETFILTER_XT_MATCH_BPF: enabled (as module)
CONFIG_INET_XFRM_MODE_TRANSPORT: missing
“ipvlan”:
CONFIG_IPVLAN: enabled (as module)
“macvlan”:
CONFIG_MACVLAN: enabled (as module)
CONFIG_DUMMY: enabled (as module)
“ftp,tftp client in container”:
CONFIG_NF_NAT_FTP: enabled (as module)
CONFIG_NF_CONNTRACK_FTP: enabled (as module)
CONFIG_NF_NAT_TFTP: enabled (as module)
CONFIG_NF_CONNTRACK_TFTP: enabled (as module)
Storage Drivers:
“btrfs”:
CONFIG_BTRFS_FS: missing
CONFIG_BTRFS_FS_POSIX_ACL: missing
“overlay”:
CONFIG_OVERLAY_FS: enabled (as module)
“zfs”:
/dev/zfs: missing
zfs command: missing
zpool command: missing
Limits:

/proc/sys/kernel/keys/root_maxkeys: 1000000
AkihiroSuda commented 10 months ago

WARNING: No cpuset support

You need to enable delegation for cpuset, but this needs systemd >= 244 (Rocky >= 9) https://rootlesscontaine.rs/getting-started/common/cgroup2/#enabling-cpu-cpuset-and-io-delegation

kdryetyln commented 7 months ago

WARNING: No cpuset support

You need to enable delegation for cpuset, but this needs systemd >= 244 (Rocky >= 9) https://rootlesscontaine.rs/getting-started/common/cgroup2/#enabling-cpu-cpuset-and-io-delegation Hello @AkihiroSuda ,

Is what you said valid also for docker rootfull? Because I cannot show the CPU and network data in dockerrootfull. However, I can see CPU and network data in Docker Stats.

AkihiroSuda commented 7 months ago

WARNING: No cpuset support

You need to enable delegation for cpuset, but this needs systemd >= 244 (Rocky >= 9) https://rootlesscontaine.rs/getting-started/common/cgroup2/#enabling-cpu-cpuset-and-io-delegation Hello @AkihiroSuda ,

Is what you said valid also for docker rootfull? Because I cannot show the CPU and network data in dockerrootfull. However, I can see CPU and network data in Docker Stats.

No, probably you are seeing a different issue. https://github.com/moby/moby/blob/master/contrib/check-config.sh might be used for analyzing the cause of the issue.

kdryetyln commented 7 months ago

I fixed my problem with this configuration:

docker run -v /:/rootfs:ro \ --name="$containerName" \ --memory=250m #{privileged} \ -v /dev/kmsg:/dev/kmsg \ -v /var/run:/var/run:rw \ -v #{dockersocketpath}/docker.sock:/var/run/docker.sock:ro \ -v /sys:/sys:ro \ -v /sys/fs/cgroup:/cgroup:ro \ -v #{dockerpath}/:/var/lib/docker:ro \ -v /dev/disk/:/dev/disk:ro \ -p #{cadvisor_ports}:8080 \ gcr.io/cadvisor/cadvisor:v0.49.1

{privileged} if docker rootfull use --privileged if its not dont use it.

{dockersocketpath} use your own dockersocketpath. It can be changed

{dockerpath} use your own docker path

If there is any wrong or excessive usage, please let me know your feedback.

PS: Previous configuration was this. I could not get my CPU and network data with this command. It didn't matter rootfull or rootless. However, the above command solved my problem in both environments.

docker run -v /:/rootfs:ro --name="containerName" --memory=250m #{privileged} -v #{dockersocketpath}/docker.sock:/var/run/docker.sock:ro -v /sys:/sys:ro -v /sys/fs/cgroup:/cgroup:ro -v #{dockerpath}/:/var/lib/docker:ro -p 8181:8080 gcr.io/cadvisor/cadvisor:v0.49.1