elastic / elastic-agent-system-metrics

Apache License 2.0
0 stars 22 forks source link

cgroups not accounting for relative paths that happen when reading from a container mount #132

Closed fearful-symmetry closed 2 months ago

fearful-symmetry commented 4 months ago

The /proc/[pid]/cgroup file contains the paths to the cgroups used by the process. Most importantly, this path is relative to the cgroup of the process checking the cgroup. From the man page:

              [3]  This field contains the pathname of the control group
                   in the hierarchy to which the process belongs.  This
                   pathname is relative to the mount point of the
                   hierarchy.

This means that if you mount in a process from the host system into a container, you get a relative path:

 docker exec -it 5b938d5468d3 /bin/bash                                                                                                                                                                                           
metricbeat@motmot:~$ cat /hostfs/proc/114856/cgroup 
0::/../elastic-agent.service

Right now this library doesn't seem to cope with this, as it assumes a universal base path:

{"log.level":"debug","@timestamp":"2024-03-08T20:32:58.467Z","log.logger":"processes","log.origin":{"file.name":"process/process.go","file.line":173},"message":"Error fetching PID info for 1023161, skipping: cgroups.GetStatsForPid: error fetching cgroupV2 controllers for cgroup location '/hostfs/sys/fs/cgroup' and path line '0::/../../user.slice/user-1000.slice/session-212.scope': open /hostfs/sys/user.slice/user-1000.slice/session-212.scope: no such file or directory","service.name":"metricbeat","ecs.version":"1.6.0"}

I'm not sure why we didn't catch this before? Perhaps there's a conflux of different docker versions, bugs and configs that have escaped most people.

fearful-symmetry commented 3 months ago

A temporary workaround is to set --cgroupns host, but that obviously has some downsides.

https://man7.org/linux/man-pages/man7/cgroup_namespaces.7.html

pierrehilbert commented 3 months ago

Blocked as required https://github.com/elastic/beats/issues/38241 first