cadvisor not exposing docker name

[Barathwaja]

Hi,

I downloaded - v0.47.0 and the query params not exposing docker name instead only container id as name. Please assist.

[rc5hack]

With v0.47.2 I have "name" label for "container_start_time_seconds" metric:

container_start_time_seconds{container_label_architecture="x86_64", container_label_build_date="hidden", container_label_com_docker_compose_config_hash="hidden", container_label_com_docker_compose_container_number="1", container_label_com_docker_compose_oneoff="False", container_label_com_docker_compose_project="minio", container_label_com_docker_compose_project_config_files="docker-compose.yaml", container_label_com_docker_compose_project_working_dir="hidden", container_label_com_docker_compose_service="minio1", container_label_com_docker_compose_version="1.25.0", container_label_description="hidden", container_label_name="MinIO", container_label_release="hidden", container_label_summary="hidden", container_label_url="https://hidden", container_label_version="hidden", id="/docker/c8d33e97e57843a2b96bcb88f2eaf71b8dcfa02f4a1757cb3b846200f4fdc4df", image="hidden", instance="hidden", job="hidden", name="minio_minio1_1"}

[Barathwaja]

@rc5hack - I'm still seeing the same even on v0.47.2. I'm not sure what is preventing me from exposing the name. Do I need to have an endpoint exposed on container because cadvisor monitor most of it without endpoint. Please assist.

docker-compose.txt

Thanks!

[rc5hack]

@Barathwaja in your docker-compose file port 8080 for cadvisor container is exposed. Could you access http://your-host:8080/metrics with web-browser (or with curl in console)? Do you see container_start_time_seconds{name="..."} in output (check it with Ctrl+F in browser or with grep in console)?

If the answer is yes, your problem is not on cadvisor side, but probably on prometheus side.

[Barathwaja]

@rc5hack Nope I don't see them on the cadvisor end.

But it picks-up other docker containers running details except not exposing name.

[rc5hack]

That's strange. Sorry, I have no other hypotheses.

[rc5hack]

Try to query on prometheus container_start_time_seconds{name!=""}?

On my environment there are 1500+container_start_time_seconds results, but only 200+ container_start_time_seconds{name!=""} results, so maybe "short" container_start_time_seconds metrics with just 1 label is not a bug while there are other "long" container_start_time_seconds metrics with "name" and other labels as well.

[Barathwaja]

Hey @rc5hack , Nope I don't have that many. Only few are there and all of them are having no names.

[rc5hack]

container_start_time_seconds{name!=""} returns nothing?

[Barathwaja]

Yes it's empty.

[Barathwaja]

Can you share your setup @rc5hack if you don't mind? TIA.

[rc5hack]

Sure! docker-compose.example.txt

[Barathwaja]

Thanks! Weird I just tried to run your cadvisor still the same :| . @rc5hack

[Barathwaja]

Hi @rc5hack, I tested on different OS (i.e Windows 11) its working; but not on MacOS Sonoma. Are you using Mac as well?

[rc5hack]

No, I don't use Mac. My hosts are Linux (Debian and Ubuntu) and Windows 11 + WSL 2 (Docker Desktop).

Maybe paths on Mac are different? I mean

      - /var/run
      - /sys
      - /var/lib/docker/
      - /dev/disk/
      - /dev/kmsg

[Barathwaja]

Okay Let me try. And by anychance you know how many days these metrics will be there in docker?

[rc5hack]

Metrics of cadvisor (http://your-host:8080/metrics) are real-time.

Metrics stored into Prometheus would be available for storage.tsdb.retention.time (default is 30 days, could be changed, see https://prometheus.io/docs/prometheus/latest/storage/#operational-aspects ).

[WaYdotNET]

some issue

[arturucraft]

No, I don't use Mac. My hosts are Linux (Debian and Ubuntu) and Windows 11 + WSL 2 (Docker Desktop).

Maybe paths on Mac are different? I mean

      - /var/run
      - /sys
      - /var/lib/docker/
      - /dev/disk/
      - /dev/kmsg

i am also using same setup, but my cadvisor is unable to detect containers. the cadvisor container keeps throwing below errors:

2024-04-23 16:09:59 E0423 12:09:59.940802       1 manager.go:1116] Failed to create existing container: /docker/b41fe27ac0154b3077980146c0e6e11ac0ee8bf27dba481fe1529c4519f89531: failed to identify the read-write layer ID for container "b41fe27ac0154b3077980146c0e6e11ac0ee8bf27dba481fe1529c4519f89531". - open /rootfs/var/lib/docker/image/overlay2/layerdb/mounts/b41fe27ac0154b3077980146c0e6e11ac0ee8bf27dba481fe1529c4519f89531/mount-id: no such file or directory
2024-04-23 16:09:59 E0423 12:09:59.948013       1 manager.go:1116] Failed to create existing container: /docker/6e7d688040e9bea6ece82c747b8dd80952ab74cb51dcfc1cb4783310d5c30836: failed to identify the read-write layer ID for container "6e7d688040e9bea6ece82c747b8dd80952ab74cb51dcfc1cb4783310d5c30836". - open /rootfs/var/lib/docker/image/overlay2/layerdb/mounts/6e7d688040e9bea6ece82c747b8dd80952ab74cb51dcfc1cb4783310d5c30836/mount-id: no such file or directory
2024-04-23 16:09:59 E0423 12:09:59.958764       1 manager.go:1116] Failed to create existing container: /docker/387ad8049239e9e27309dc96ed3a6773fc6c088ca51788396e6832f1f7927311: failed to identify the read-write layer ID for container "387ad8049239e9e27309dc96ed3a6773fc6c088ca51788396e6832f1f7927311". - open /rootfs/var/lib/docker/image/overlay2/layerdb/mounts/387ad8049239e9e27309dc96ed3a6773fc6c088ca51788396e6832f1f7927311/mount-id: no such file or directory
2024-04-23 16:09:59 E0423 12:09:59.964707       1 manager.go:1116] Failed to create existing container: /docker/0319c7c02809137d989f982a246829232a8035c814a00d96257a77ccb2b1e976: failed to identify the read-write layer ID for container "0319c7c02809137d989f982a246829232a8035c814a00d96257a77ccb2b1e976". - open /rootfs/var/lib/docker/image/overlay2/layerdb/mounts/0319c7c02809137d989f982a246829232a8035c814a00d96257a77ccb2b1e976/mount-id: no such file or directory

i have followed this link without luck. uninstalled and installed back the desktop docker and still no luck.

my cadvisor section of docker compose file looks like below:

  cadvisor:
    #image: google/cadvisor:latest
    image: gcr.io/cadvisor/cadvisor:v0.49.1
    container_name: monitoring_cadvisor
    restart: unless-stopped
    volumes:
      - /:/rootfs:ro
      - /var/run:/var/run:rw
      - /sys:/sys:ro
      - /var/lib/docker/:/var/lib/docker:ro
      - /dev/disk/:/dev/disk:ro
        #- /mnt/my-docker/version-pack-data/community/docker:/rootfs/var/lib/docker:ro
      - /etc/machine-id:/etc/machine-id:ro
      - /dev/disk:/dev/disk:ro
      - /etc/localtime:/etc/localtime:ro
      - /etc/timezone:/etc/timezone:ro
    command:
      - --docker_only=true
    privileged: true
    devices:
      - /dev/kmsg:/dev/kmsg
    expose:
      - 8080
    ports:
      - 8080:8080
    networks:
      - loki

any hints are appreciated.

TIA