Open Barathwaja opened 10 months ago
With v0.47.2 I have "name" label for "container_start_time_seconds" metric:
container_start_time_seconds{container_label_architecture="x86_64", container_label_build_date="hidden", container_label_com_docker_compose_config_hash="hidden", container_label_com_docker_compose_container_number="1", container_label_com_docker_compose_oneoff="False", container_label_com_docker_compose_project="minio", container_label_com_docker_compose_project_config_files="docker-compose.yaml", container_label_com_docker_compose_project_working_dir="hidden", container_label_com_docker_compose_service="minio1", container_label_com_docker_compose_version="1.25.0", container_label_description="hidden", container_label_name="MinIO", container_label_release="hidden", container_label_summary="hidden", container_label_url="https://hidden", container_label_version="hidden", id="/docker/c8d33e97e57843a2b96bcb88f2eaf71b8dcfa02f4a1757cb3b846200f4fdc4df", image="hidden", instance="hidden", job="hidden", name="minio_minio1_1"}
@rc5hack - I'm still seeing the same even on v0.47.2. I'm not sure what is preventing me from exposing the name. Do I need to have an endpoint exposed on container because cadvisor monitor most of it without endpoint. Please assist.
Thanks!
@Barathwaja in your docker-compose file port 8080
for cadvisor container is exposed. Could you access http://your-host:8080/metrics
with web-browser (or with curl in console)? Do you see container_start_time_seconds{name="..."}
in output (check it with Ctrl
+F
in browser or with grep
in console)?
If the answer is yes, your problem is not on cadvisor side, but probably on prometheus side.
@rc5hack Nope I don't see them on the cadvisor end.
But it picks-up other docker containers running details except not exposing name.
That's strange. Sorry, I have no other hypotheses.
Try to query on prometheus container_start_time_seconds{name!=""}
?
On my environment there are 1500+container_start_time_seconds
results, but only 200+ container_start_time_seconds{name!=""}
results, so maybe "short" container_start_time_seconds metrics with just 1 label is not a bug while there are other "long" container_start_time_seconds metrics with "name" and other labels as well.
Hey @rc5hack , Nope I don't have that many. Only few are there and all of them are having no names.
container_start_time_seconds{name!=""}
returns nothing?
Yes it's empty.
Can you share your setup @rc5hack if you don't mind? TIA.
Thanks! Weird I just tried to run your cadvisor
still the same :| . @rc5hack
Hi @rc5hack, I tested on different OS (i.e Windows 11) its working; but not on MacOS Sonoma. Are you using Mac as well?
No, I don't use Mac. My hosts are Linux (Debian and Ubuntu) and Windows 11 + WSL 2 (Docker Desktop).
Maybe paths on Mac are different? I mean
- /var/run
- /sys
- /var/lib/docker/
- /dev/disk/
- /dev/kmsg
Okay Let me try. And by anychance you know how many days these metrics will be there in docker?
Metrics of cadvisor (http://your-host:8080/metrics
) are real-time.
Metrics stored into Prometheus would be available for storage.tsdb.retention.time
(default is 30 days, could be changed, see https://prometheus.io/docs/prometheus/latest/storage/#operational-aspects ).
some issue
No, I don't use Mac. My hosts are Linux (Debian and Ubuntu) and Windows 11 + WSL 2 (Docker Desktop).
Maybe paths on Mac are different? I mean
- /var/run - /sys - /var/lib/docker/ - /dev/disk/ - /dev/kmsg
i am also using same setup, but my cadvisor is unable to detect containers. the cadvisor container keeps throwing below errors:
2024-04-23 16:09:59 E0423 12:09:59.940802 1 manager.go:1116] Failed to create existing container: /docker/b41fe27ac0154b3077980146c0e6e11ac0ee8bf27dba481fe1529c4519f89531: failed to identify the read-write layer ID for container "b41fe27ac0154b3077980146c0e6e11ac0ee8bf27dba481fe1529c4519f89531". - open /rootfs/var/lib/docker/image/overlay2/layerdb/mounts/b41fe27ac0154b3077980146c0e6e11ac0ee8bf27dba481fe1529c4519f89531/mount-id: no such file or directory
2024-04-23 16:09:59 E0423 12:09:59.948013 1 manager.go:1116] Failed to create existing container: /docker/6e7d688040e9bea6ece82c747b8dd80952ab74cb51dcfc1cb4783310d5c30836: failed to identify the read-write layer ID for container "6e7d688040e9bea6ece82c747b8dd80952ab74cb51dcfc1cb4783310d5c30836". - open /rootfs/var/lib/docker/image/overlay2/layerdb/mounts/6e7d688040e9bea6ece82c747b8dd80952ab74cb51dcfc1cb4783310d5c30836/mount-id: no such file or directory
2024-04-23 16:09:59 E0423 12:09:59.958764 1 manager.go:1116] Failed to create existing container: /docker/387ad8049239e9e27309dc96ed3a6773fc6c088ca51788396e6832f1f7927311: failed to identify the read-write layer ID for container "387ad8049239e9e27309dc96ed3a6773fc6c088ca51788396e6832f1f7927311". - open /rootfs/var/lib/docker/image/overlay2/layerdb/mounts/387ad8049239e9e27309dc96ed3a6773fc6c088ca51788396e6832f1f7927311/mount-id: no such file or directory
2024-04-23 16:09:59 E0423 12:09:59.964707 1 manager.go:1116] Failed to create existing container: /docker/0319c7c02809137d989f982a246829232a8035c814a00d96257a77ccb2b1e976: failed to identify the read-write layer ID for container "0319c7c02809137d989f982a246829232a8035c814a00d96257a77ccb2b1e976". - open /rootfs/var/lib/docker/image/overlay2/layerdb/mounts/0319c7c02809137d989f982a246829232a8035c814a00d96257a77ccb2b1e976/mount-id: no such file or directory
i have followed this link without luck. uninstalled and installed back the desktop docker and still no luck.
my cadvisor section of docker compose file looks like below:
cadvisor:
#image: google/cadvisor:latest
image: gcr.io/cadvisor/cadvisor:v0.49.1
container_name: monitoring_cadvisor
restart: unless-stopped
volumes:
- /:/rootfs:ro
- /var/run:/var/run:rw
- /sys:/sys:ro
- /var/lib/docker/:/var/lib/docker:ro
- /dev/disk/:/dev/disk:ro
#- /mnt/my-docker/version-pack-data/community/docker:/rootfs/var/lib/docker:ro
- /etc/machine-id:/etc/machine-id:ro
- /dev/disk:/dev/disk:ro
- /etc/localtime:/etc/localtime:ro
- /etc/timezone:/etc/timezone:ro
command:
- --docker_only=true
privileged: true
devices:
- /dev/kmsg:/dev/kmsg
expose:
- 8080
ports:
- 8080:8080
networks:
- loki
any hints are appreciated.
TIA
Hi,
I downloaded - v0.47.0 and the query params not exposing docker name instead only container id as name. Please assist.