search

google / cadvisor

Analyzes resource usage and performance characteristics of running containers.
Other
17.25k stars 2.33k forks source link

Does rootless podman works? Can someone help me with setup? #3500

Open frenkye opened 8 months ago

frenkye commented 8 months ago

Hi,

i tried to setup this for 3 scenarios:

  1. docker root - works
  2. docker rootless - works
  3. podman rootles - do not work

I have issue with cadvisor not returning metrics about cpu, memory of running containers. I would appreciate, if anyone could give me some hint how to do it.

I know for long time there wasnt full support in 0.47.2, but this should be now in 0.49.1 available since support was added in 0.48.0

Client:       Podman Engine
Version:      4.9.3
API Version:  4.9.3
Go Version:   go1.21.6
Built:        Thu Jan  1 01:00:00 1970
OS/Arch:      linux/amd64

Last run command:

        RUN_IMAGE="podman run -d --restart always -p ${PORT}:8080 \
            -v /:/rootfs:ro \
            -v /etc/machine-id:/etc/machine-id:ro \
            -v /sys:/sys:ro \
            -v /dev/disk:/dev/disk:ro \
            -v /run/containerd/containerd.sock:/run/containerd/containerd.sock:rw \
            -v /run/user/${UID}/podman/podman.sock:/var/run/podman/podman.sock:ro \
            -v ${ROOTLESS_HOMEDIR}/.local/share/containers/:/var/lib/containers:ro \
            --name=${CONTAINER_NAME} \
            gcr.io/cadvisor/cadvisor:${CONTAINER_VERSION}"
mrgleeco commented 8 months ago

I am also trying rootless and having no luck seeing container data. Additionally i can add i am ensuring that the podman flag is set, ie. --podman="unix:///var/run/podman/podman.sock" and that i do see ack in logging, eg.

I0319 02:23:08.841905       1 plugin.go:89] Registering Podman factory
I0319 02:23:08.886522       1 factory.go:221] Registration of the podman container factory successfully
Cub0n commented 8 months ago

In my environment it runs as Rootless: https://github.com/Cub0n/Grafana-Prometheus-cAdvisor

Aldenar commented 2 months ago

I am having issues despite having tried the last post's suggestions.

My environment:

OS: Debian Bookworm 
Kernel: v6.1.0-25-amd64
Container provider: Rootless-podman v4.3.1

And my run command:

podman run -d \
    --name cadvisor -p 6252:8080 \
    --volume=/:/rootfs:ro \
        --volume=/etc/machine-id:/etc/machine-id:ro \
    --volume=$HOME/.local/share/containers:/rootfs/var/lib/containers:ro \
    --volume=/run/user/$(id -u)/podman:/var/run/podman:ro \
    --volume=/sys:/sys:ro \
    --volume=/dev/disk:/dev/disk:ro \ 
    --device=/dev/kmesg \
    --privileged \
    --restart=always \
    gcr.io/cadvisor/cadvisor:v0.50.0 --podman="unix:///var/run/podman/podman.sock" --housekeeping_interval=10s

In the container log, I can see that the podman engine is clearly recognized:

I0924 14:12:06.543932       1 plugin.go:89] Registering Podman factory
I0924 14:12:07.225363       1 factory.go:221] Registration of the podman container factory successfully

However, once cAdvisor finishes booting, it periodically emits messages such as:

I0924 14:12:07.229351       1 fs.go:465] unable to determine file system type, partition mountpoint does not exist: /rootfs/var/lib/containers/storage/overlay-containers/bcb56a2590cd18df4f6d150af2a22039b7e8573dda427684c1f30c2a6305f18f/userdata/shm
I0924 14:12:07.229424       1 fs.go:465] unable to determine file system type, partition mountpoint does not exist: /rootfs/var/lib/containers/storage/vfs/dir/12f87d333a84c414a3213120cd7ad298f061e8ad908388c8f4f147656337d183/rootfs/var/lib/containers/storage/overlay/b28195bda9c0841825291d20d872686d3468b2e4604399174238a1893711e2f6/merged

At the web UI's /podman/ URL, cAdvisor v0.50.0 is able to display both, Podman's version and the API Version (While the v0.49.1 release just showed "Unknown" for both), making me think that cAdvisor itself is able to communicate with Podman fine, but issues arise somewhere down the processing line instead.

Out of desperation, I tried mounting the user dbus socket ($XDG_RUNTIME_DIR/bus) into the container as well, no change.

Also tried mounting $HOME/.local/share/containers/storage and $HOME/.local/share/containers to /rootfs/var/lib/containers (Due to the stat errors), no change in either case.

Full log (In case it was needed, verbosity set to 10 [-v 10]):

I0924 15:14:59.828337       1 cadvisor.go:123] enabled metrics: app,cpu,cpuLoad,disk,diskIO,memory,network,oom_event,percpu,perf_event
I0924 15:14:59.828407       1 storagedriver.go:55] Caching stats in memory for 2m0s
I0924 15:14:59.828946       1 plugin.go:69] Docker not connected: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
I0924 15:15:04.833081       1 plugin.go:41] CRI-O not connected: Get "http://%2Fvar%2Frun%2Fcrio%2Fcrio.sock/info": dial unix /var/run/crio/crio.sock: connect: no such file or directory
I0924 15:15:04.835956       1 fs.go:135] Filesystem UUIDs: map[74f8691d-6ef6-4203-bf96-d8d6281bad84:/dev/vdc 88ccd220-990f-4fdd-966d-1e38e02840c2:/dev/vda1 e479a1ea-c096-4894-831b-ee5ea1c4c6e7:/dev/vdb]
I0924 15:15:04.836005       1 fs.go:136] Filesystem partitions: map[/dev:{mountpoint:/dev major:0 minor:59 fsType:tmpfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:56 fsType:tmpfs blockSize:0} /dev/vdb:{mountpoint:/rootfs major:254 minor:16 fsType:ext4 blockSize:0} /dev/vdc:{mountpoint:/ major:254 minor:32 fsType:ext4 blockSize:0} /etc/hostname:{mountpoint:/etc/hostname major:0 minor:42 fsType:tmpfs blockSize:0} /etc/hosts:{mountpoint:/etc/hosts major:0 minor:42 fsType:tmpfs blockSize:0} /etc/resolv.conf:{mountpoint:/etc/resolv.conf major:0 minor:42 fsType:tmpfs blockSize:0} /rootfs/dev/shm:{mountpoint:/rootfs/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /rootfs/home/rootless-podman/.local/share/containers/storage/vfs-containers/24881dbbd8a3e90287b9bd1e633d14dad7a62c50678f764ff643879e400031e5/userdata/shm:{mountpoint:/rootfs/home/rootless-podman/.local/share/containers/storage/vfs-containers/24881dbbd8a3e90287b9bd1e633d14dad7a62c50678f764ff643879e400031e5/userdata/shm major:0 minor:56 fsType:tmpfs blockSize:0} /rootfs/home/rootless-podman/.local/share/containers/storage/vfs/dir/932d3b0c183ac5aa36c8668cfbd84c9a22b0db9fc1c6f13ac6b08bdec1124ae2/dev:{mountpoint:/rootfs/home/rootless-podman/.local/share/containers/storage/vfs/dir/932d3b0c183ac5aa36c8668cfbd84c9a22b0db9fc1c6f13ac6b08bdec1124ae2/dev major:0 minor:59 fsType:tmpfs blockSize:0} /rootfs/run:{mountpoint:/rootfs/run major:0 minor:23 fsType:tmpfs blockSize:0} /rootfs/run/lock:{mountpoint:/rootfs/run/lock major:0 minor:25 fsType:tmpfs blockSize:0} /rootfs/run/netns:{mountpoint:/rootfs/run/netns major:0 minor:23 fsType:tmpfs blockSize:0} /rootfs/run/user/1001:{mountpoint:/rootfs/run/user/1001 major:0 minor:41 fsType:tmpfs blockSize:0} /rootfs/run/user/1002:{mountpoint:/rootfs/run/user/1002 major:0 minor:42 fsType:tmpfs blockSize:0} /rootfs/run/user/1002/netns:{mountpoint:/rootfs/run/user/1002/netns major:0 minor:42 fsType:tmpfs blockSize:0} /rootfs/run/user/10063:{mountpoint:/rootfs/run/user/10063 major:0 minor:55 fsType:tmpfs blockSize:0} /rootfs/tmp:{mountpoint:/rootfs/tmp major:0 minor:37 fsType:tmpfs blockSize:0} /rootfs/var/lib/containers/storage/overlay-containers/bcb56a2590cd18df4f6d150af2a22039b7e8573dda427684c1f30c2a6305f18f/userdata/shm:{mountpoint:/rootfs/var/lib/containers/storage/overlay-containers/bcb56a2590cd18df4f6d150af2a22039b7e8573dda427684c1f30c2a6305f18f/userdata/shm major:0 minor:44 fsType:tmpfs blockSize:0} /rootfs/var/lib/containers/vfs-containers/24881dbbd8a3e90287b9bd1e633d14dad7a62c50678f764ff643879e400031e5/userdata/shm:{mountpoint:/rootfs/var/lib/containers/vfs-containers/24881dbbd8a3e90287b9bd1e633d14dad7a62c50678f764ff643879e400031e5/userdata/shm major:0 minor:56 fsType:tmpfs blockSize:0} /rootfs/var/lib/containers/vfs/dir/932d3b0c183ac5aa36c8668cfbd84c9a22b0db9fc1c6f13ac6b08bdec1124ae2/dev:{mountpoint:/rootfs/var/lib/containers/vfs/dir/932d3b0c183ac5aa36c8668cfbd84c9a22b0db9fc1c6f13ac6b08bdec1124ae2/dev major:0 minor:59 fsType:tmpfs blockSize:0} /rootfs/var/lib/containers/vfs/dir/932d3b0c183ac5aa36c8668cfbd84c9a22b0db9fc1c6f13ac6b08bdec1124ae2/dev/shm:{mountpoint:/rootfs/var/lib/containers/vfs/dir/932d3b0c183ac5aa36c8668cfbd84c9a22b0db9fc1c6f13ac6b08bdec1124ae2/dev/shm major:0 minor:56 fsType:tmpfs blockSize:0} /rootfs/var/lib/containers/vfs/dir/932d3b0c183ac5aa36c8668cfbd84c9a22b0db9fc1c6f13ac6b08bdec1124ae2/etc/hostname:{mountpoint:/rootfs/var/lib/containers/vfs/dir/932d3b0c183ac5aa36c8668cfbd84c9a22b0db9fc1c6f13ac6b08bdec1124ae2/etc/hostname major:0 minor:42 fsType:tmpfs blockSize:0} /rootfs/var/lib/containers/vfs/dir/932d3b0c183ac5aa36c8668cfbd84c9a22b0db9fc1c6f13ac6b08bdec1124ae2/etc/hosts:{mountpoint:/rootfs/var/lib/containers/vfs/dir/932d3b0c183ac5aa36c8668cfbd84c9a22b0db9fc1c6f13ac6b08bdec1124ae2/etc/hosts major:0 minor:42 fsType:tmpfs blockSize:0} /rootfs/var/lib/containers/vfs/dir/932d3b0c183ac5aa36c8668cfbd84c9a22b0db9fc1c6f13ac6b08bdec1124ae2/etc/resolv.conf:{mountpoint:/rootfs/var/lib/containers/vfs/dir/932d3b0c183ac5aa36c8668cfbd84c9a22b0db9fc1c6f13ac6b08bdec1124ae2/etc/resolv.conf major:0 minor:42 fsType:tmpfs blockSize:0} /rootfs/var/lib/containers/vfs/dir/932d3b0c183ac5aa36c8668cfbd84c9a22b0db9fc1c6f13ac6b08bdec1124ae2/rootfs/dev/shm:{mountpoint:/rootfs/var/lib/containers/vfs/dir/932d3b0c183ac5aa36c8668cfbd84c9a22b0db9fc1c6f13ac6b08bdec1124ae2/rootfs/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /rootfs/var/lib/containers/vfs/dir/932d3b0c183ac5aa36c8668cfbd84c9a22b0db9fc1c6f13ac6b08bdec1124ae2/rootfs/home/rootless-podman/.local/share/containers/storage/vfs-containers/24881dbbd8a3e90287b9bd1e633d14dad7a62c50678f764ff643879e400031e5/userdata/shm:{mountpoint:/rootfs/var/lib/containers/vfs/dir/932d3b0c183ac5aa36c8668cfbd84c9a22b0db9fc1c6f13ac6b08bdec1124ae2/rootfs/home/rootless-podman/.local/share/containers/storage/vfs-containers/24881dbbd8a3e90287b9bd1e633d14dad7a62c50678f764ff643879e400031e5/userdata/shm major:0 minor:56 fsType:tmpfs blockSize:0} /rootfs/var/lib/containers/vfs/dir/932d3b0c183ac5aa36c8668cfbd84c9a22b0db9fc1c6f13ac6b08bdec1124ae2/rootfs/home/rootless-podman/.local/share/containers/storage/vfs/dir/932d3b0c183ac5aa36c8668cfbd84c9a22b0db9fc1c6f13ac6b08bdec1124ae2/dev:{mountpoint:/rootfs/var/lib/containers/vfs/dir/932d3b0c183ac5aa36c8668cfbd84c9a22b0db9fc1c6f13ac6b08bdec1124ae2/rootfs/home/rootless-podman/.local/share/containers/storage/vfs/dir/932d3b0c183ac5aa36c8668cfbd84c9a22b0db9fc1c6f13ac6b08bdec1124ae2/dev major:0 minor:59 fsType:tmpfs blockSize:0} /rootfs/var/lib/containers/vfs/dir/932d3b0c183ac5aa36c8668cfbd84c9a22b0db9fc1c6f13ac6b08bdec1124ae2/rootfs/run:{mountpoint:/rootfs/var/lib/containers/vfs/dir/932d3b0c183ac5aa36c8668cfbd84c9a22b0db9fc1c6f13ac6b08bdec1124ae2/rootfs/run major:0 minor:23 fsType:tmpfs blockSize:0} /rootfs/var/lib/containers/vfs/dir/932d3b0c183ac5aa36c8668cfbd84c9a22b0db9fc1c6f13ac6b08bdec1124ae2/rootfs/run/lock:{mountpoint:/rootfs/var/lib/containers/vfs/dir/932d3b0c183ac5aa36c8668cfbd84c9a22b0db9fc1c6f13ac6b08bdec1124ae2/rootfs/run/lock major:0 minor:25 fsType:tmpfs blockSize:0} /rootfs/var/lib/containers/vfs/dir/932d3b0c183ac5aa36c8668cfbd84c9a22b0db9fc1c6f13ac6b08bdec1124ae2/rootfs/run/netns:{mountpoint:/rootfs/var/lib/containers/vfs/dir/932d3b0c183ac5aa36c8668cfbd84c9a22b0db9fc1c6f13ac6b08bdec1124ae2/rootfs/run/netns major:0 minor:23 fsType:tmpfs blockSize:0} /rootfs/var/lib/containers/vfs/dir/932d3b0c183ac5aa36c8668cfbd84c9a22b0db9fc1c6f13ac6b08bdec1124ae2/rootfs/run/user/1001:{mountpoint:/rootfs/var/lib/containers/vfs/dir/932d3b0c183ac5aa36c8668cfbd84c9a22b0db9fc1c6f13ac6b08bdec1124ae2/rootfs/run/user/1001 major:0 minor:41 fsType:tmpfs blockSize:0} /rootfs/var/lib/containers/vfs/dir/932d3b0c183ac5aa36c8668cfbd84c9a22b0db9fc1c6f13ac6b08bdec1124ae2/rootfs/run/user/1002:{mountpoint:/rootfs/var/lib/containers/vfs/dir/932d3b0c183ac5aa36c8668cfbd84c9a22b0db9fc1c6f13ac6b08bdec1124ae2/rootfs/run/user/1002 major:0 minor:42 fsType:tmpfs blockSize:0} /rootfs/var/lib/containers/vfs/dir/932d3b0c183ac5aa36c8668cfbd84c9a22b0db9fc1c6f13ac6b08bdec1124ae2/rootfs/run/user/1002/netns:{mountpoint:/rootfs/var/lib/containers/vfs/dir/932d3b0c183ac5aa36c8668cfbd84c9a22b0db9fc1c6f13ac6b08bdec1124ae2/rootfs/run/user/1002/netns major:0 minor:42 fsType:tmpfs blockSize:0} /rootfs/var/lib/containers/vfs/dir/932d3b0c183ac5aa36c8668cfbd84c9a22b0db9fc1c6f13ac6b08bdec1124ae2/rootfs/run/user/10063:{mountpoint:/rootfs/var/lib/containers/vfs/dir/932d3b0c183ac5aa36c8668cfbd84c9a22b0db9fc1c6f13ac6b08bdec1124ae2/rootfs/run/user/10063 major:0 minor:55 fsType:tmpfs blockSize:0} /rootfs/var/lib/containers/vfs/dir/932d3b0c183ac5aa36c8668cfbd84c9a22b0db9fc1c6f13ac6b08bdec1124ae2/rootfs/tmp:{mountpoint:/rootfs/var/lib/containers/vfs/dir/932d3b0c183ac5aa36c8668cfbd84c9a22b0db9fc1c6f13ac6b08bdec1124ae2/rootfs/tmp major:0 minor:37 fsType:tmpfs blockSize:0} /rootfs/var/lib/containers/vfs/dir/932d3b0c183ac5aa36c8668cfbd84c9a22b0db9fc1c6f13ac6b08bdec1124ae2/rootfs/var/lib/containers/storage/overlay-containers/bcb56a2590cd18df4f6d150af2a22039b7e8573dda427684c1f30c2a6305f18f/userdata/shm:{mountpoint:/rootfs/var/lib/containers/vfs/dir/932d3b0c183ac5aa36c8668cfbd84c9a22b0db9fc1c6f13ac6b08bdec1124ae2/rootfs/var/lib/containers/storage/overlay-containers/bcb56a2590cd18df4f6d150af2a22039b7e8573dda427684c1f30c2a6305f18f/userdata/shm major:0 minor:44 fsType:tmpfs blockSize:0} /rootfs/var/lib/containers/vfs/dir/932d3b0c183ac5aa36c8668cfbd84c9a22b0db9fc1c6f13ac6b08bdec1124ae2/run/.containerenv:{mountpoint:/rootfs/var/lib/containers/vfs/dir/932d3b0c183ac5aa36c8668cfbd84c9a22b0db9fc1c6f13ac6b08bdec1124ae2/run/.containerenv major:0 minor:42 fsType:tmpfs blockSize:0} /rootfs/var/lib/containers/vfs/dir/932d3b0c183ac5aa36c8668cfbd84c9a22b0db9fc1c6f13ac6b08bdec1124ae2/run/bus:{mountpoint:/rootfs/var/lib/containers/vfs/dir/932d3b0c183ac5aa36c8668cfbd84c9a22b0db9fc1c6f13ac6b08bdec1124ae2/run/bus major:0 minor:42 fsType:tmpfs blockSize:0} /rootfs/var/lib/containers/vfs/dir/932d3b0c183ac5aa36c8668cfbd84c9a22b0db9fc1c6f13ac6b08bdec1124ae2/run/podman:{mountpoint:/rootfs/var/lib/containers/vfs/dir/932d3b0c183ac5aa36c8668cfbd84c9a22b0db9fc1c6f13ac6b08bdec1124ae2/run/podman major:0 minor:42 fsType:tmpfs blockSize:0} /rootfs/var/lib/containers/vfs/dir/932d3b0c183ac5aa36c8668cfbd84c9a22b0db9fc1c6f13ac6b08bdec1124ae2/sys/fs/cgroup:{mountpoint:/rootfs/var/lib/containers/vfs/dir/932d3b0c183ac5aa36c8668cfbd84c9a22b0db9fc1c6f13ac6b08bdec1124ae2/sys/fs/cgroup major:0 minor:62 fsType:tmpfs blockSize:0} /run/.containerenv:{mountpoint:/run/.containerenv major:0 minor:42 fsType:tmpfs blockSize:0} /run/bus:{mountpoint:/run/bus major:0 minor:42 fsType:tmpfs blockSize:0} /run/podman:{mountpoint:/run/podman major:0 minor:42 fsType:tmpfs blockSize:0} /sys/fs/cgroup:{mountpoint:/sys/fs/cgroup major:0 minor:62 fsType:tmpfs blockSize:0} overlay_0-45:{mountpoint:/rootfs/var/lib/containers/vfs/dir/932d3b0c183ac5aa36c8668cfbd84c9a22b0db9fc1c6f13ac6b08bdec1124ae2/rootfs/var/lib/containers/storage/overlay/4ebab359a22d4bc661da84854aa60fe01cf21024b295a7c88c911ca9be6ba3b6/merged major:0 minor:45 fsType:overlay blockSize:0}]
I0924 15:15:04.837242       1 fs.go:465] unable to determine file system type, partition mountpoint does not exist: /rootfs/var/lib/containers/storage/overlay-containers/bcb56a2590cd18df4f6d150af2a22039b7e8573dda427684c1f30c2a6305f18f/userdata/shm
I0924 15:15:04.837303       1 fs.go:465] unable to determine file system type, partition mountpoint does not exist: /rootfs/var/lib/containers/vfs/dir/932d3b0c183ac5aa36c8668cfbd84c9a22b0db9fc1c6f13ac6b08bdec1124ae2/rootfs/var/lib/containers/storage/overlay/4ebab359a22d4bc661da84854aa60fe01cf21024b295a7c88c911ca9be6ba3b6/merged
I0924 15:15:04.837567       1 fs.go:465] unable to determine file system type, partition mountpoint does not exist: /rootfs/var/lib/containers/vfs/dir/932d3b0c183ac5aa36c8668cfbd84c9a22b0db9fc1c6f13ac6b08bdec1124ae2/rootfs/var/lib/containers/storage/overlay-containers/bcb56a2590cd18df4f6d150af2a22039b7e8573dda427684c1f30c2a6305f18f/userdata/shm
I0924 15:15:04.842496       1 manager.go:217] Machine: {Timestamp:2024-09-24 15:15:04.842094781 +0000 UTC m=+5.032745492 CPUVendorID:GenuineIntel NumCores:4 NumPhysicalCores:4 NumSockets:1 CpuFrequency:2399996 MemoryCapacity:8326438912 SwapCapacity:1070571520 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:558c49448e5948de855b168f1f10a017 SystemUUID:558c49448e5948de855b168f1f10a017 BootID:21a4a216-4670-44bd-bf38-74ff4e1cc10e Filesystems:[{Device:/run/podman DeviceMajor:0 DeviceMinor:42 Capacity:832643072 Type:vfs Inodes:203282 HasInodes:true} {Device:/sys/fs/cgroup DeviceMajor:0 DeviceMinor:62 Capacity:0 Type:vfs Inodes:0 HasInodes:true} {Device:/rootfs/var/lib/containers/vfs/dir/932d3b0c183ac5aa36c8668cfbd84c9a22b0db9fc1c6f13ac6b08bdec1124ae2/rootfs/run/lock DeviceMajor:0 DeviceMinor:25 Capacity:5242880 Type:vfs Inodes:1016411 HasInodes:true} {Device:/rootfs/var/lib/containers/vfs/dir/932d3b0c183ac5aa36c8668cfbd84c9a22b0db9fc1c6f13ac6b08bdec1124ae2/rootfs/run/user/1002 DeviceMajor:0 DeviceMinor:42 Capacity:832643072 Type:vfs Inodes:203282 HasInodes:true} {Device:/rootfs/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:4163219456 Type:vfs Inodes:1016411 HasInodes:true} {Device:/rootfs/run/user/1002/netns DeviceMajor:0 DeviceMinor:42 Capacity:832643072 Type:vfs Inodes:203282 HasInodes:true} {Device:/rootfs/tmp DeviceMajor:0 DeviceMinor:37 Capacity:832647168 Type:vfs Inodes:1016411 HasInodes:true} {Device:/etc/hosts DeviceMajor:0 DeviceMinor:42 Capacity:832643072 Type:vfs Inodes:203282 HasInodes:true} {Device:/rootfs/var/lib/containers/vfs/dir/932d3b0c183ac5aa36c8668cfbd84c9a22b0db9fc1c6f13ac6b08bdec1124ae2/rootfs/run/netns DeviceMajor:0 DeviceMinor:23 Capacity:832647168 Type:vfs Inodes:1016411 HasInodes:true} {Device:/rootfs/var/lib/containers/vfs/dir/932d3b0c183ac5aa36c8668cfbd84c9a22b0db9fc1c6f13ac6b08bdec1124ae2/rootfs/home/rootless-podman/.local/share/containers/storage/vfs-containers/24881dbbd8a3e90287b9bd1e633d14dad7a62c50678f764ff643879e400031e5/userdata/shm DeviceMajor:0 DeviceMinor:56 Capacity:65536000 Type:vfs Inodes:1016411 HasInodes:true} {Device:/rootfs/var/lib/containers/vfs/dir/932d3b0c183ac5aa36c8668cfbd84c9a22b0db9fc1c6f13ac6b08bdec1124ae2/rootfs/home/rootless-podman/.local/share/containers/storage/vfs/dir/932d3b0c183ac5aa36c8668cfbd84c9a22b0db9fc1c6f13ac6b08bdec1124ae2/dev DeviceMajor:0 DeviceMinor:59 Capacity:67108864 Type:vfs Inodes:1016411 HasInodes:true} {Device:/rootfs/var/lib/containers/vfs/dir/932d3b0c183ac5aa36c8668cfbd84c9a22b0db9fc1c6f13ac6b08bdec1124ae2/dev/shm DeviceMajor:0 DeviceMinor:56 Capacity:65536000 Type:vfs Inodes:1016411 HasInodes:true} {Device:/rootfs/var/lib/containers/vfs/dir/932d3b0c183ac5aa36c8668cfbd84c9a22b0db9fc1c6f13ac6b08bdec1124ae2/etc/hostname DeviceMajor:0 DeviceMinor:42 Capacity:832643072 Type:vfs Inodes:203282 HasInodes:true} {Device:/rootfs/run/lock DeviceMajor:0 DeviceMinor:25 Capacity:5242880 Type:vfs Inodes:1016411 HasInodes:true} {Device:/rootfs/run/user/1002 DeviceMajor:0 DeviceMinor:42 Capacity:832643072 Type:vfs Inodes:203282 HasInodes:true} {Device:/run/.containerenv DeviceMajor:0 DeviceMinor:42 Capacity:832643072 Type:vfs Inodes:203282 HasInodes:true} {Device:/rootfs/var/lib/containers/vfs/dir/932d3b0c183ac5aa36c8668cfbd84c9a22b0db9fc1c6f13ac6b08bdec1124ae2/sys/fs/cgroup DeviceMajor:0 DeviceMinor:62 Capacity:0 Type:vfs Inodes:0 HasInodes:true} {Device:/rootfs/run/user/1001 DeviceMajor:0 DeviceMinor:41 Capacity:832643072 Type:vfs Inodes:203282 HasInodes:true} {Device:/rootfs/var/lib/containers/vfs/dir/932d3b0c183ac5aa36c8668cfbd84c9a22b0db9fc1c6f13ac6b08bdec1124ae2/rootfs/run/user/10063 DeviceMajor:0 DeviceMinor:55 Capacity:832643072 Type:vfs Inodes:203282 HasInodes:true} {Device:/rootfs/var/lib/containers/vfs/dir/932d3b0c183ac5aa36c8668cfbd84c9a22b0db9fc1c6f13ac6b08bdec1124ae2/etc/hosts DeviceMajor:0 DeviceMinor:42 Capacity:832643072 Type:vfs Inodes:203282 HasInodes:true} {Device:/rootfs/var/lib/containers/vfs/dir/932d3b0c183ac5aa36c8668cfbd84c9a22b0db9fc1c6f13ac6b08bdec1124ae2/etc/resolv.conf DeviceMajor:0 DeviceMinor:42 Capacity:832643072 Type:vfs Inodes:203282 HasInodes:true} {Device:/rootfs/var/lib/containers/vfs/dir/932d3b0c183ac5aa36c8668cfbd84c9a22b0db9fc1c6f13ac6b08bdec1124ae2/run/podman DeviceMajor:0 DeviceMinor:42 Capacity:832643072 Type:vfs Inodes:203282 HasInodes:true} {Device:/dev/vdc DeviceMajor:254 DeviceMinor:32 Capacity:10464022528 Type:vfs Inodes:655360 HasInodes:true} {Device:/dev DeviceMajor:0 DeviceMinor:59 Capacity:67108864 Type:vfs Inodes:1016411 HasInodes:true} {Device:/etc/resolv.conf DeviceMajor:0 DeviceMinor:42 Capacity:832643072 Type:vfs Inodes:203282 HasInodes:true} {Device:/run/bus DeviceMajor:0 DeviceMinor:42 Capacity:832643072 Type:vfs Inodes:203282 HasInodes:true} {Device:/rootfs/var/lib/containers/vfs/dir/932d3b0c183ac5aa36c8668cfbd84c9a22b0db9fc1c6f13ac6b08bdec1124ae2/rootfs/tmp DeviceMajor:0 DeviceMinor:37 Capacity:832647168 Type:vfs Inodes:1016411 HasInodes:true} {Device:/rootfs/run/netns DeviceMajor:0 DeviceMinor:23 Capacity:832647168 Type:vfs Inodes:1016411 HasInodes:true} {Device:/rootfs/run/user/10063 DeviceMajor:0 DeviceMinor:55 Capacity:832643072 Type:vfs Inodes:203282 HasInodes:true} {Device:/rootfs/home/rootless-podman/.local/share/containers/storage/vfs/dir/932d3b0c183ac5aa36c8668cfbd84c9a22b0db9fc1c6f13ac6b08bdec1124ae2/dev DeviceMajor:0 DeviceMinor:59 Capacity:67108864 Type:vfs Inodes:1016411 HasInodes:true} {Device:/rootfs/var/lib/containers/storage/overlay-containers/bcb56a2590cd18df4f6d150af2a22039b7e8573dda427684c1f30c2a6305f18f/userdata/shm DeviceMajor:0 DeviceMinor:44 Capacity:0 Type: Inodes:0 HasInodes:false} {Device:/rootfs/var/lib/containers/vfs/dir/932d3b0c183ac5aa36c8668cfbd84c9a22b0db9fc1c6f13ac6b08bdec1124ae2/run/.containerenv DeviceMajor:0 DeviceMinor:42 Capacity:832643072 Type:vfs Inodes:203282 HasInodes:true} {Device:/rootfs/var/lib/containers/vfs/dir/932d3b0c183ac5aa36c8668cfbd84c9a22b0db9fc1c6f13ac6b08bdec1124ae2/run/bus DeviceMajor:0 DeviceMinor:42 Capacity:832643072 Type:vfs Inodes:203282 HasInodes:true} {Device:overlay_0-45 DeviceMajor:0 DeviceMinor:45 Capacity:0 Type: Inodes:0 HasInodes:false} {Device:/rootfs/var/lib/containers/vfs-containers/24881dbbd8a3e90287b9bd1e633d14dad7a62c50678f764ff643879e400031e5/userdata/shm DeviceMajor:0 DeviceMinor:56 Capacity:65536000 Type:vfs Inodes:1016411 HasInodes:true} {Device:/rootfs/var/lib/containers/vfs/dir/932d3b0c183ac5aa36c8668cfbd84c9a22b0db9fc1c6f13ac6b08bdec1124ae2/dev DeviceMajor:0 DeviceMinor:59 Capacity:67108864 Type:vfs Inodes:1016411 HasInodes:true} {Device:/rootfs/var/lib/containers/vfs/dir/932d3b0c183ac5aa36c8668cfbd84c9a22b0db9fc1c6f13ac6b08bdec1124ae2/rootfs/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:4163219456 Type:vfs Inodes:1016411 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:56 Capacity:65536000 Type:vfs Inodes:1016411 HasInodes:true} {Device:/rootfs/var/lib/containers/vfs/dir/932d3b0c183ac5aa36c8668cfbd84c9a22b0db9fc1c6f13ac6b08bdec1124ae2/rootfs/run DeviceMajor:0 DeviceMinor:23 Capacity:832647168 Type:vfs Inodes:1016411 HasInodes:true} {Device:/rootfs/var/lib/containers/vfs/dir/932d3b0c183ac5aa36c8668cfbd84c9a22b0db9fc1c6f13ac6b08bdec1124ae2/rootfs/run/user/1002/netns DeviceMajor:0 DeviceMinor:42 Capacity:832643072 Type:vfs Inodes:203282 HasInodes:true} {Device:/dev/vdb DeviceMajor:254 DeviceMinor:16 Capacity:33501757440 Type:vfs Inodes:2097152 HasInodes:true} {Device:/rootfs/run DeviceMajor:0 DeviceMinor:23 Capacity:832647168 Type:vfs Inodes:1016411 HasInodes:true} {Device:/rootfs/home/rootless-podman/.local/share/containers/storage/vfs-containers/24881dbbd8a3e90287b9bd1e633d14dad7a62c50678f764ff643879e400031e5/userdata/shm DeviceMajor:0 DeviceMinor:56 Capacity:65536000 Type:vfs Inodes:1016411 HasInodes:true} {Device:/etc/hostname DeviceMajor:0 DeviceMinor:42 Capacity:832643072 Type:vfs Inodes:203282 HasInodes:true} {Device:/rootfs/var/lib/containers/vfs/dir/932d3b0c183ac5aa36c8668cfbd84c9a22b0db9fc1c6f13ac6b08bdec1124ae2/rootfs/run/user/1001 DeviceMajor:0 DeviceMinor:41 Capacity:832643072 Type:vfs Inodes:203282 HasInodes:true} {Device:/rootfs/var/lib/containers/vfs/dir/932d3b0c183ac5aa36c8668cfbd84c9a22b0db9fc1c6f13ac6b08bdec1124ae2/rootfs/var/lib/containers/storage/overlay-containers/bcb56a2590cd18df4f6d150af2a22039b7e8573dda427684c1f30c2a6305f18f/userdata/shm DeviceMajor:0 DeviceMinor:44 Capacity:0 Type: Inodes:0 HasInodes:false}] DiskMap:map[254:0:{Name:vda Major:254 Minor:0 Size:1073741824 Scheduler:none} 254:16:{Name:vdb Major:254 Minor:16 Size:34359738368 Scheduler:none} 254:32:{Name:vdc Major:254 Minor:32 Size:10737418240 Scheduler:none} 254:48:{Name:vdd Major:254 Minor:48 Size:10737418240 Scheduler:none}] NetworkDevices:[{Name:cni-podman0 MacAddress:fe:b0:5f:5d:cd:d3 Speed:10000 Mtu:1500} {Name:ens18 MacAddress:ea:ce:a2:6f:f5:6d Speed:-1 Mtu:1500}] Topology:[{Id:0 Memory:8326438912 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:4194304 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:4194304 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:4194304 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:4194304 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:16777216 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
I0924 15:15:04.842796       1 manager.go:226] Cannot gather resctrl metrics: unable to initialize resctrl: Intel RDT not available
I0924 15:15:04.842835       1 manager.go:233] Version: {KernelVersion:6.1.0-25-amd64 ContainerOsVersion:Alpine Linux v3.18 DockerVersion: DockerAPIVersion: CadvisorVersion:v0.50.0 CadvisorRevision:256737f3}
I0924 15:15:04.842963       1 factory.go:55] Registering systemd factory
I0924 15:15:04.842977       1 factory.go:221] Registration of the systemd container factory successfully
I0924 15:15:04.843212       1 factory.go:219] Registration of the crio container factory failed: Get "http://%2Fvar%2Frun%2Fcrio%2Fcrio.sock/info": dial unix /var/run/crio/crio.sock: connect: no such file or directory
I0924 15:15:04.844238       1 factory.go:219] Registration of the mesos container factory failed: unable to create mesos agent client: failed to get version
I0924 15:15:04.844575       1 factory.go:219] Registration of the docker container factory failed: failed to validate Docker info: failed to detect Docker info: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
I0924 15:15:05.110871       1 plugin.go:89] Registering Podman factory
I0924 15:15:05.814754       1 factory.go:221] Registration of the podman container factory successfully
I0924 15:15:05.814900       1 factory.go:219] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
I0924 15:15:05.814995       1 factory.go:103] Registering Raw factory
I0924 15:15:05.815014       1 manager.go:1196] Started watching for new ooms in manager
W0924 15:15:05.815114       1 manager.go:306] Could not configure a source for OOM detection, disabling OOM events: open /dev/kmsg: operation not permitted
I0924 15:15:05.815190       1 factory.go:45] / not handled by systemd handler
I0924 15:15:05.815206       1 factory.go:279] Factory "systemd" was unable to handle container "/"
I0924 15:15:05.815217       1 factory.go:279] Factory "podman" was unable to handle container "/"
I0924 15:15:05.815225       1 factory.go:275] Using factory "raw" for container "/"
I0924 15:15:05.815996       1 manager.go:981] Added container: "/" (aliases: [], namespace: "")
I0924 15:15:05.816284       1 handler.go:325] Added event &{/ 2024-09-24 15:14:59.773821837 +0000 UTC containerCreation {<nil>}}
I0924 15:15:05.816499       1 manager.go:319] Starting recovery of all containers
I0924 15:15:05.816616       1 manager.go:324] Recovery completed
I0924 15:15:05.816659       1 container.go:529] Start housekeeping for container "/"
I0924 15:15:05.817066       1 cadvisor.go:176] Starting cAdvisor version: v0.50.0-256737f3 on port 8080
I0924 15:15:05.819660       1 fs.go:465] unable to determine file system type, partition mountpoint does not exist: /rootfs/var/lib/containers/storage/overlay-containers/bcb56a2590cd18df4f6d150af2a22039b7e8573dda427684c1f30c2a6305f18f/userdata/shm
I0924 15:15:05.819826       1 fs.go:465] unable to determine file system type, partition mountpoint does not exist: /rootfs/var/lib/containers/vfs/dir/932d3b0c183ac5aa36c8668cfbd84c9a22b0db9fc1c6f13ac6b08bdec1124ae2/rootfs/var/lib/containers/storage/overlay/4ebab359a22d4bc661da84854aa60fe01cf21024b295a7c88c911ca9be6ba3b6/merged
I0924 15:15:05.820021       1 fs.go:465] unable to determine file system type, partition mountpoint does not exist: /rootfs/var/lib/containers/vfs/dir/932d3b0c183ac5aa36c8668cfbd84c9a22b0db9fc1c6f13ac6b08bdec1124ae2/rootfs/var/lib/containers/storage/overlay-containers/bcb56a2590cd18df4f6d150af2a22039b7e8573dda427684c1f30c2a6305f18f/userdata/shm
I0924 15:15:12.386198       1 podman.go:137] Request took 710.265128ms
I0924 15:15:12.854067       1 containers.go:245] Request took 974.14µs

Any suggestions on making Rootless Podman work would be much appreciated!

jollySleeper commented 5 days ago

Hi, @Aldenar the path for $HOME/.local/share/containers should point to /var/lib/containers instead of /rootfs/var/lib/containers. Try this way and let me know. Also, you can try with --volume /var/lib/containers:/var/lib/containers:ro. See if it helps.

Aldenar commented 4 days ago

Hi, @Aldenar the path for $HOME/.local/share/containers should point to /var/lib/containers instead of /rootfs/var/lib/containers. Try this way and let me know. Also, you can try with --volume /var/lib/containers:/var/lib/containers:ro. See if it helps.

Unfortunately, that didn't help either. I tried deploying the latest v0.51.0, and under /podman/, I get some information like the OS, Podman and API versions. But nothing regarding the specific containers running:

Driver Status
    Version 4.3.1
    API Version 1.41
    Kernel Version 6.1.0-27-amd64
    OS Version debian
    Host Name podman-test-1
    Root Directory /home/www/test.local/.local/share/containers/storage
    Execution Driver
    Number of Images 2
    Number of Containers 1
    Storage
        Driver vfs

Though the container's log does indicate it registered the Podman API correctly

I1126 13:22:19.575080       1 plugin.go:89] Registering Podman factory
I1126 13:22:20.297581       1 factory.go:221] Registration of the podman container factory successfully

Run command:

podman run -d --restart always -p 6800:8080 \
-v /:/rootfs:ro \
-v /etc/machine-id:/etc/machine-id:ro \
-v /dev/disk:/dev/disk:ro \
-v /sys:/sys:ro \
-v /dev/kmsg:/dev/kmsg:ro \
-v ${HOME}/.local/share/containers:/var/lib/containers:ro \
-v /run/user/${UID}/podman/podman.sock:/var/run/podman/podman.sock:ro \
gcr.io/cadvisor/cadvisor:v0.51.0
jollySleeper commented 3 days ago

I tried deploying the latest v0.51.0

How, afaik latest version is v0.49.1 right?

So, you mean you are getting the info about everything including podman too but not the container names right? Can you just check this out #3628 once because I think we may be having the same issues.

Also, could you try adding cgroup to the container using -v /sys/fs/cgroup:/sys/fs/cgroup:ro. Please let me know.

frenkye commented 3 days ago

How, afaik latest version is v0.49.1 right?

@jollySleeper you can't trust latest on cAdvisor project. There was issue, where they start ignoring updating latest tag. We discused it already here https://github.com/google/cadvisor/issues/3363

jollySleeper commented 3 days ago

Thanks for the info @frenkye . Will check that issue.

jollySleeper commented 3 days ago

@Aldenar I updated to v0.51.0 and tried it. No errors have been thrown out to me in logs but the /podman/ route isn't working for me now.

The Output of /podman/ route :

failed to get podman info: unable to parse docker host `unix://`

Fraction of cadvisor logs:

I1127 11:18:08.987241       1 manager.go:233] Version: {KernelVersion:6.1.43-rockchip-rk3588 ContainerOsVersion:Alpine Linux v3.18 DockerVersion: DockerAPIVersion: CadvisorVersion:v0.51.0 CadvisorRevision:255cff78}
I1127 11:18:08.987613       1 factory.go:55] Registering systemd factory
I1127 11:18:08.987644       1 factory.go:221] Registration of the systemd container factory successfully
I1127 11:18:08.988050       1 factory.go:219] Registration of the crio container factory failed: Get "http://%2Fvar%2Frun%2Fcrio%2Fcrio.sock/info": dial unix /var/run/crio/crio.sock: connect: no such file or directory
I1127 11:18:08.990602       1 factory.go:219] Registration of the mesos container factory failed: unable to create mesos agent client: failed to get version
I1127 11:18:08.990754       1 factory.go:219] Registration of the docker container factory failed: unable to communicate with docker daemon: unable to parse docker host `unix://`
I1127 11:18:09.487221       1 plugin.go:89] Registering Podman factory
I1127 11:18:16.911611       1 factory.go:221] Registration of the podman container factory successfully
I1127 11:18:16.912394       1 factory.go:219] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
I1127 11:18:16.912825       1 factory.go:103] Registering Raw factory
I1127 11:18:16.913072       1 manager.go:1196] Started watching for new ooms in manager
I1127 11:18:16.919055       1 manager.go:319] Starting recovery of all containers
I1127 11:18:16.919728       1 manager.go:324] Recovery completed
I1127 11:18:16.921409       1 cadvisor.go:176] Starting cAdvisor version: v0.51.0-255cff78 on port 8080