Podman-docker

[usbport11]

Problem: cAdvisor not see container nor in docker nor in podman.

System

[root@cadviser-podman /]# hostnamectl
   Static hostname: cadviser-podman
         Icon name: computer-vm
           Chassis: vm
        Machine ID: 43e46d0e4ae24963b040c022d0b8e514
           Boot ID: e2aeb7a0f0ad41aeaced19f77e31e238
    Virtualization: oracle
  Operating System: Oracle Linux Server 8.9
       CPE OS Name: cpe:/o:oracle:linux:8:9:server
            Kernel: Linux 5.4.17-2136.329.3.1.el8uek.x86_64
      Architecture: x86-64

cAdvisor standalone binary 0.49.1

[root@cadviser-podman /]# cat /usr/lib/systemd/system/cadvisor.service
[Unit]
Description=cAdvisor
Wants=network-online.target
After=network-online.target
[Service]
User=root
Group=root
Type=simple
#Restart=on-failure
ExecStart=/usr/bin/cadvisor
[Install]
WantedBy=multi-user.target

Steps to reproduce

[root@cadviser-podman /]# dnf install podman

[root@cadviser-podman /]# docker ps -a
Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg.
CONTAINER ID  IMAGE       COMMAND     CREATED     STATUS      PORTS       NAMES

[root@cadviser-podman /]# docker create --name=test alpine:latest /bin/sh -c "while true; do echo hello world; sleep 1; done"
Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg.
e93d554c34dc1d90f17f367a30a5d861ef07e8b86752822182a173485a289a6a

[root@cadviser-podman /]# docker start test
Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg.
test

[root@cadviser-podman /]# docker ps -a
Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg.
CONTAINER ID  IMAGE                            COMMAND               CREATED         STATUS        PORTS       NAMES
e93d554c34dc  docker.io/library/alpine:latest  /bin/sh -c while ...  57 seconds ago  Up 9 seconds              test

[root@cadviser-podman /]# systemctl restart cadvisor

[root@cadviser-podman /]# systemctl status cadvisor
● cadvisor.service - cAdvisor
   Loaded: loaded (/usr/lib/systemd/system/cadvisor.service; disabled; vendor preset: disabled)
   Active: active (running) since Mon 2024-04-01 21:41:47 MSK; 45s ago
 Main PID: 7628 (cadvisor)
    Tasks: 9 (limit: 4064)
   Memory: 30.1M
   CGroup: /system.slice/cadvisor.service
           └─7628 /usr/bin/cadvisor

Apr 01 21:41:49 cadviser-podman cadvisor[7628]: I0401 21:41:49.361772    7628 manager.go:1196] Started watching for new ooms in manager
Apr 01 21:41:49 cadviser-podman cadvisor[7628]: I0401 21:41:49.362712    7628 manager.go:319] Starting recovery of all containers
Apr 01 21:41:49 cadviser-podman cadvisor[7628]: E0401 21:41:49.415768    7628 manager.go:1116] Failed to create existing container: /machine.slice/libpod-conmon-e93d554c34dc1d90f17f367a30a5d861ef07e8b86752822182a173485a289a6a.scope: failed to identify the read-write layer ID for container "e93d554c34dc1d90f17f367a30a5d861ef07e8b86752822182a173485a289a6a". - open /var/lib/containers/storage/image/overlay/layerdb/mounts/e93d554c34dc1d90f17f367a30a5d861ef07e8b86752822182a173485a289a6a/mount-id: no such file or directory
Apr 01 21:41:49 cadviser-podman cadvisor[7628]: E0401 21:41:49.429480    7628 manager.go:1116] Failed to create existing container: /machine.slice/libpod-e93d554c34dc1d90f17f367a30a5d861ef07e8b86752822182a173485a289a6a.scope: failed to identify the read-write layer ID for container "e93d554c34dc1d90f17f367a30a5d861ef07e8b86752822182a173485a289a6a". - open /var/lib/containers/storage/image/overlay/layerdb/mounts/e93d554c34dc1d90f17f367a30a5d861ef07e8b86752822182a173485a289a6a/mount-id: no such file or directory
Apr 01 21:41:49 cadviser-podman cadvisor[7628]: I0401 21:41:49.553318    7628 manager.go:324] Recovery completed
Apr 01 21:41:49 cadviser-podman cadvisor[7628]: E0401 21:41:49.648486    7628 manager.go:1116] Failed to create existing container: /machine.slice/libpod-conmon-e93d554c34dc1d90f17f367a30a5d861ef07e8b86752822182a173485a289a6a.scope: failed to identify the read-write layer ID for container "e93d554c34dc1d90f17f367a30a5d861ef07e8b86752822182a173485a289a6a". - open /var/lib/containers/storage/image/overlay/layerdb/mounts/e93d554c34dc1d90f17f367a30a5d861ef07e8b86752822182a173485a289a6a/mount-id: no such file or directory
Apr 01 21:41:49 cadviser-podman cadvisor[7628]: E0401 21:41:49.661902    7628 manager.go:1116] Failed to create existing container: /machine.slice/libpod-e93d554c34dc1d90f17f367a30a5d861ef07e8b86752822182a173485a289a6a.scope: failed to identify the read-write layer ID for container "e93d554c34dc1d90f17f367a30a5d861ef07e8b86752822182a173485a289a6a". - open /var/lib/containers/storage/image/overlay/layerdb/mounts/e93d554c34dc1d90f17f367a30a5d861ef07e8b86752822182a173485a289a6a/mount-id: no such file or directory
Apr 01 21:41:49 cadviser-podman cadvisor[7628]: I0401 21:41:49.662256    7628 cadvisor.go:176] Starting cAdvisor version: v0.49.1-6f3f25ba on port 8080
Apr 01 21:41:49 cadviser-podman cadvisor[7628]: W0401 21:41:49.675351    7628 manager.go:1169] Failed to process watch event {EventType:0 Name:/machine.slice/libpod-conmon-e93d554c34dc1d90f17f367a30a5d861ef07e8b86752822182a173485a289a6a.scope WatchSource:0}: failed to identify the read-write layer ID for container "e93d554c34dc1d90f17f367a30a5d861ef07e8b86752822182a173485a289a6a". - open /var/lib/containers/storage/image/overlay/layerdb/mounts/e93d554c34dc1d90f17f367a30a5d861ef07e8b86752822182a173485a289a6a/mount-id: no such file or directory
Apr 01 21:41:49 cadviser-podman cadvisor[7628]: W0401 21:41:49.680255    7628 manager.go:1169] Failed to process watch event {EventType:0 Name:/machine.slice/libpod-e93d554c34dc1d90f17f367a30a5d861ef07e8b86752822182a173485a289a6a.scope WatchSource:0}: failed to identify the read-write layer ID for container "e93d554c34dc1d90f17f367a30a5d861ef07e8b86752822182a173485a289a6a". - open /var/lib/containers/storage/image/overlay/layerdb/mounts/e93d554c34dc1d90f17f367a30a5d861ef07e8b86752822182a173485a289a6a/mount-id: no such file or directory

Please help.

[usbport11]

Create folders mkdir -p /var/lib/containers/storage/image/overlay/layerdb/mounts/e93d554c34dc1d90f17f367a30a5d861ef07e8b86752822182a173485a289a6a

mkdir -p /var/lib/containers/storage/containers/e93d554c34dc1d90f17f367a30a5d861ef07e8b86752822182a173485a289a6a

mkdir -p /var/lib/containers/storage/overlay/upper

Create file > /var/lib/containers/storage/image/overlay/layerdb/mounts/e93d554c34dc1d90f17f367a30a5d861ef07e8b86752822182a173485a289a6a/mount-id

Restart

systemctl restart cadvisor

Container appear in cAdvisor. But now in prometheus after query there are two containers with same name.

Query container_cpu_usage_seconds_total{name!=""}

Result

container_cpu_usage_seconds_total{cpu="cpu00", id="/machine.slice/libpod-conmon-e93d554c34dc1d90f17f367a30a5d861ef07e8b86752822182a173485a289a6a.scope", image="docker.io/library/alpine:latest", instance="192.168.1.209:8080", job="cadviser", name="test"}
0.14341915
container_cpu_usage_seconds_total{cpu="cpu00", id="/machine.slice/libpod-e93d554c34dc1d90f17f367a30a5d861ef07e8b86752822182a173485a289a6a.scope", image="docker.io/library/alpine:latest", instance="192.168.1.209:8080", job="cadviser", name="test"}
1.872301843

[Crapshit]

Could you please check if this helps you?: https://github.com/google/cadvisor/issues/3421#issuecomment-2449725212