Container fails at start on Synology with Prometheus

[Raptor05]

Hello,

I've installed cAdvisor following the tutorial from mariushosting to show the docker resources in grafana.

Here is the service part for cAdvisor from my docker-stack:

version: "3.9"
services:
  grafana:
   ...

  prometheus:
   ...

  node-exporter:
    ...

  snmp-exporter:
   ...

  cadvisor:
    image: gcr.io/cadvisor/cadvisor:latest
    command:
      - '--docker_only=true'
    container_name: Prometheus-cAdvisor
    hostname: prometheus-cadvisor
    networks:
      - prometheus-net
    mem_limit: 256m
    mem_reservation: 64m
    cpu_shares: 512
    security_opt:
      - no-new-privileges=true
    read_only: true
    volumes:
      - /:/rootfs:ro
      - /var/run:/var/run:ro
      - /sys:/sys:ro
      - /var/run/docker.sock:/var/run/docker.sock:ro
    restart: on-failure:5

networks:
  grafana-net:
    name: grafana-net
    ipam:
      config:
        - subnet: 192.168.50.0/24
  prometheus-net:
    name: prometheus-net
    ipam:
      config:
        - subnet: 192.168.51.0/24

All other containers are starting, the grafana dashboard is working, but the diagrams for the docker containers are empty, since the cAdvisor crashed in the background.

Here is an extract from the logs:

W0116 15:02:00.744061       1 fs.go:208] stat failed on /dev/mapper/cachedev_0 with error: no such file or directory
W0116 15:02:00.750904       1 sysinfo.go:203] Nodes topology is not available, providing CPU topology
W0116 15:02:00.751333       1 sysfs.go:348] unable to read /sys/devices/system/cpu/cpu0/online: open /sys/devices/system/cpu/cpu0/online: no such file or directory
W0116 15:02:00.813282       1 manager.go:288] Could not configure a source for OOM detection, disabling OOM events: open /dev/kmsg: no such file or directory
W0116 15:02:00.813837       1 manager.go:949] Error getting perf_event cgroup path: "could not find path for resource \"perf_event\" for container \"/\""
W0116 15:02:00.865423       1 manager.go:949] Error getting perf_event cgroup path: "could not find path for resource \"perf_event\" for container \"/docker/57445f9f121aaa673a42b4fd50d638823ef1dfbf25e54a34f1bb9e0917f49e96\""
W0116 15:02:00.871683       1 manager.go:949] Error getting perf_event cgroup path: "could not find path for resource \"perf_event\" for container \"/docker/d0de522296b8b88e45f1dc98d5f845a77ff5f03eef4395adeae1f36e61b452e4\""
W0116 15:02:00.876601       1 manager.go:949] Error getting perf_event cgroup path: "could not find path for resource \"perf_event\" for container \"/docker/1dc98202dab365519a92cbc82675cf851aa69970fec6ba01eb0582265ad94368\""
...
F0116 15:02:01.001852       1 cadvisor.go:204] Failed to start manager: inotify_add_watch /sys/fs/cgroup/cpuset: no space left on device
goroutine 1 [running]:
k8s.io/klog/v2.stacks(0xc0000d2001, 0xc00018a280, 0x89, 0x248)
    /go/pkg/mod/k8s.io/klog/v2@v2.2.0/klog.go:996 +0xb8
....

Do you have any idea on how to solve that issue?

Thanks in advance! 🙂

Note: The solutions from #3436 and #1846 are not working for me.

[xaviergxf]

not working for me either

[johnwyles]

Yep. Months later and I can't seem to find any place that has an answer either. All lead to threads like this one with a bunch of dead ends. If I did find one though there are so many now that I would sprint back through all the threads and comment on all of them to save all of our sufferings for the betterment of our shared futures.

[iwankgb]

Unfortunately I have no Synology hardware at hand and I can't help you :(