search

google / cadvisor

Analyzes resource usage and performance characteristics of running containers.
Other
17.12k stars 2.32k forks source link

container_tasks_state giving result of 0 for every element #2739

Open david-bickford opened 3 years ago

david-bickford commented 3 years ago

I'm trying to view the status of our Docker containers in Prometheus using cAdvisor. Every other stat seems to be working except for container_tasks_state. I'm running this on an AWS EC2 instance running Linux. Here is the version info output from cAdvisor. cadvisor_version_info{cadvisorRevision="65fa5b44",cadvisorVersion="v0.37.0",dockerVersion="19.03.2",instance="host:8080",job="cadvisor",kernelVersion="4.15.0-1051-aws",osVersion="Alpine Linux v3.12"}

This is a link to screenshots of the results I get back from the container_tasks_state query and the running containers in our EC2 instance. As you can see the values are all 0 even though they are running.

I'm deploying cAdvisor to our EC2 instances using Ansible instead of running a docker-compose since we have many EC2 instances. Below is the Ansible script which pulls and runs cAdvisor.

- name: pulling and starting cAdvisor
  become: yes
  docker_container:
    name: "cadvisor"
    detach: True
    privileged: True
    networks:
      - name: "my-net"
    image: gcr.io/cadvisor/cadvisor:v0.37.0
    command: ["--enable_load_reader=true", "--container_hints=/etc/cadvisor/container_hints.json", "--v=4", "--stderrthreshold=0", "--log_dir=/var/log", "--logtostderr=false", "
--alsologtostderr=false"]
    state: started
    restart_policy: "{{ restart_policy }}"
    env:
      NODE_ENV: "{{ NODE_ENV }}"
    volumes:
      - "{{ cert_vol }}:/opt/scribe/cert"
      - /:/rootfs:ro
      - /var/run:/var/run:rw
      - /sys:/sys:ro
      - /var/lib/docker/:/var/lib/docker:ro
      - /dev/disk/:/dev/disk:ro
    log_driver: syslog
    log_options: tag={{ service_name }}/'{{ '{{' }}.ID{{ '}}' }}'
    published_ports: 0.0.0.0:8080:8080

Things I've tried:

ChaoHsupin commented 3 years ago

I also encountered a similar problem. Have you solved it please？ #2841

iwankgb commented 3 years ago

Quick code analysis suggests that you need to set -enable_load_reader flag.

sgpinkus commented 2 years ago

What value are you expecting? Isn't is supposed to be 0?

UlyC commented 2 years ago

Same issue.

Things I've tried:

  1. add the flag --pid=host
  2. add the flag --volume=/cgroup:/cgroup:ro
  3. add the flag --userns=host
  4. setting --enable_load_reader=true
  5. putting both containers on the same docker network, or use 'network_mode: "host"'
  6. testing different versions of cAdvisor (I tested like 8 other versions)

Reference Links：