search

google / cadvisor

Analyzes resource usage and performance characteristics of running containers.
Other
17.01k stars 2.31k forks source link

Cadvisor does not collect metrics when podman-docker is installed #3554

Open rushilenekar20 opened 3 months ago

rushilenekar20 commented 3 months ago

System:

[root@machine /]# hostnamectl
   Static hostname:  machine
         Icon name: computer-server
           Chassis: server
        Machine ID: 3342db489c014533b01ac7ebf64a59e0
           Boot ID: cffcc4cf0a9141dd871335c015f0650a
  Operating System: Red Hat Enterprise Linux 8.8 (Ootpa)
       CPE OS Name: cpe:/o:redhat:enterprise_linux:8::baseos
            Kernel: Linux 4.18.0-477.55.1.el8_8.x86_64
      Architecture: x86-64
 [root@machine /]# cadvisor --version
cAdvisor version v0.49.1 (6f3f25ba)

Problem: Cadvisor does not collect metrics when podman-docker is installed # rpm -qa | grep podman-docker podman-docker-4.4.1-19.module+el8.8.0+21741+dd5be1e7.noarch

This package installs a script named docker that emulates the Docker CLI by executes podman commands, it also creates links between all Docker CLI man pages and podman.

When tried to run cadvisor ( v0.49.1 ) binary it returns with following error

I0624 10:09:20.158369  669512 factory.go:352] Registering Docker factory
I0624 10:09:20.158396  669512 factory.go:221] Registration of the docker container factory successfully
I0624 10:09:20.595080  669512 plugin.go:89] Registering Podman factory
W0624 10:09:22.391362  669512 plugin.go:105] Podman rootless containers not working with cgroups v1!
I0624 10:09:22.391386  669512 factory.go:221] Registration of the podman container factory successfully
.
.
E0624 10:14:12.424102  684886 manager.go:1116] Failed to create existing container: /machine.slice/machine-libpod_pod_f97d9e5122f0e6941b5064c07338676adc6e973319abdf7e7c2b28a6b74debaa.slice/libpod-13e0ee6ba4eb38d8b31a9afdfbc72a1529c9b841adaf1b1c232f9e7f7cdd2f92.scope: failed to identify the read-write layer ID for container "13e0ee6ba4eb38d8b31a9afdfbc72a1529c9b841adaf1b1c232f9e7f7cdd2f92". - open /var/lib/containers/storage/image/overlay/layerdb/mounts/13e0ee6ba4eb38d8b31a9afdfbc72a1529c9b841adaf1b1c232f9e7f7cdd2f92/mount-id: no such file or directory
E0624 10:14:12.439989  684886 manager.go:1116] Failed to create existing container: /machine.slice/libpod-1618ef4de6fafc7b4695d00d8eb993d9e5890fae72d5af79eb42f29635ca13e7.scope: failed to identify the read-write layer ID for container "1618ef4de6fafc7b4695d00d8eb993d9e5890fae72d5af79eb42f29635ca13e7". - open /var/lib/containers/storage/image/overlay/layerdb/mounts/1618ef4de6fafc7b4695d00d8eb993d9e5890fae72d5af79eb42f29635ca13e7/mount-id: no such file or directory

After further debugging, found docker factory tried to read - I0619 12:30:17.191319 3651243 factory.go:279] Using factory "docker" for container "/machine.slice/13e0ee6ba4eb38d8b31a9afdfbc72a1529c9b841adaf1b1c232f9e7f7cdd2f92.service"

Solution: When we used podman-docker package, system has unix socket path for both podman and docker /run/docker.sock /run/podman/podman.sock Hence, cadvisor was able to call both podman and docker APIs, and registers factory for both.

I0624 10:25:21.191734  720196 factory.go:352] Registering Docker factory
I0624 10:25:21.191757  720196 factory.go:221] Registration of the docker container factory successfully
I0624 10:25:21.631911  720196 plugin.go:89] Registering Podman factory
W0624 10:25:23.453891  720196 plugin.go:105] Podman rootless containers not working with cgroups v1!
I0624 10:25:23.453911  720196 factory.go:221] Registration of the podman container factory successfully

_When tried cadvisor running with only podman unix socket (by giving empty value to flag for docker unix socket path) - ./cadvisor-v0.49.1-linux-amd64 --podman=unix:///run/podman/podman.sock --docker=unix://

I0624 10:28:50.554944  731305 factory.go:219] Registration of the docker container factory failed: failed to validate Docker info: failed to detect Docker info: error during connect: Get "http:///v1.24/info": http: no Host in request URL
I0624 10:28:51.154913  731305 plugin.go:89] Registering Podman factory
W0624 10:28:51.789149  731305 plugin.go:105] Podman rootless containers not working with cgroups v1!
I0624 10:28:51.789168  731305 factory.go:221] Registration of the podman container factory successfully

Cadvisor is working as expected

  1. No above error - Failed to create existing container:...
  2. Collected data for containers with image and name 
    
#  curl http://localhost:8080/metrics  | grep container_cpu_system_seconds_total
container_cpu_system_seconds_total{container_label_architecture="x86_64",container_label_build_date="2024-06-06T00:48:40",container_label_com_redhat_component="ubi9-...",container_label_io_k8s_display_name="Red Hat Universal Base Image 9",container_label_io_openshift_expose_services="",container_label_io_openshift_tags="base rhel9",container_label_maintainer="Red Hat, Inc.",container_label_name="ubi9",container_label_release="1123",container_label_summary="Provides the latest release of Red Hat Universal Base Image 9.",container_label_url="https://access.redhat.com/containers/#/registry.access.redhat.com/ubi9/images/9.4-1123",container_label_vcs_ref="92a4a475241865d0d11bd861fb2b29fbd9b17df0",container_label_vcs_type="git",container_label_vendor="Red Hat, Inc.",container_label_version="9.4",id="/machine.slice/libpod-49385cbc66da72004fe10721bfc6fd3cda9d74cf317a015e404800ca137637e6.scope",image="registry.access.redhat.com/ubi9/ubi:latest",name="serene_blackburn"} 1.54 1719225219156

podman ps

CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 49385cbc66da registry.access.redhat.com/ubi9/ubi:latest /bin/bash 10 days ago Up 10 days ago serene_blackburn



@dashpole @Creatone @iwankgb 
Is it valid workaround to use cadvisor binary( v0.49.1) for podman-docker system by not registering docker factory by giving empty/wrong unix socket path to docker?