Multiple Metrics die down in just 5 minutes

[jayjoshi64]

When I start the container, all the data is collected correctly. Slowly one by one the container start disappearing. Within 5 minutes, all the containers are usually gone from the graphs.

These are the counters which are affected:

• container_cpu_usage_seconds_total
• container_network_receive_bytes_total
• container_network_transmit_bytes_total

My setup:

• Running through docker compose in an ubuntu machine.
• My docker compose:

  cadvisor:
  image: gcr.io/cadvisor/cadvisor:latest
  container_name: cadvisor
  command:
    - '-port=8098'
    - '--global_housekeeping_interval=10m0s'
    - '--housekeeping_interval=5m0s'
    - '--max_housekeeping_interval=10m0s'
  ports:
    - 8098:8098
  volumes:
    - /:/rootfs:ro
    - /var/run:/var/run:ro
    - /sys:/sys:ro
    - /var/lib/docker/:/var/lib/docker:ro
    - /dev/disk/:/dev/disk:ro
  devices:
    - /dev/kmsg
  networks:
    - console
  privileged: true
  restart: unless-stopped

[chengjoey]

could you please provide cadvisor logs

[jayjoshi64]

Looks like the issue was with scrape_interval. Previously I set it to 30s. but after reducing it to 15s, the issue seems to be resolved. I don't know the reason behind it though.

I also tried following a way to reduce resources from https://github.com/google/cadvisor/issues/2523 which might have helped resolve the issue as well.

edit: one thing I noticed during my own troubleshooting is that the metrics are still available. they just didn't have correct container_name attached. most of the metric started having empty container names. that was the main reason grafana stopped showing them on the dashboard.