CPU of cadvisor too high

ramuskay commented 4 years ago

When i deploy the stack with default values the cadvisor container had a CPU usage too high regards to regular containers. Maybe its' normal ?

Expected Behaviour

I would expect that CPU usage was like other containers not that much

Current Behaviour

Bellow the CPU taken by cadvisor container

The top of my server :

Possible Solution

I tried to follow the case : https://github.com/google/cadvisor/issues/1774 But it didn't seems to work on my server

Steps to Reproduce (for bugs)

Provide a link to a live example, or steps to reproduce this bug. Include code to reproduce, if relevant:

git clone https://github.com/vegasbrianc/prometheus.git
cd prometheus
docker stack deploy -c docker-stack.yml prom

Your Environment

[root@xxxxx prometheus]# docker version
Client: Docker Engine - Community
 Version:           19.03.7
 API version:       1.40
 Go version:        go1.12.17
 Git commit:        7141c199a2
 Built:             Wed Mar  4 01:24:10 2020
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.7
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.17
  Git commit:       7141c199a2
  Built:            Wed Mar  4 01:22:45 2020
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.2.6
  GitCommit:        894b81a4b802e4eb2a91d1ce216b8817763c29fb
 runc:
  Version:          1.0.0-rc8
  GitCommit:        425e105d5a03fabd737a126ad93d62a9eeede87f
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

My OS is CentOS Linux release 8.1.1911 (Core)

github-actions[bot] commented 4 years ago

Awesome! Thanks for taking the time to open an issue. We will have a look and answer as soon as we can.' first issue

immortaly007 commented 4 years ago

I'm experiencing the same issue (15% CPU usage here). What helped a bit was putting the scrape_interval for cadvisor to 15s in prometheus/prometheus.yml:

  - job_name: 'cadvisor'

    # Override the global default and scrape targets from this job every 5 seconds.
-    scrape_interval: 5s
+    scrape_interval: 15s

    dns_sd_configs:
    - names:
      - 'tasks.cadvisor'
      type: 'A'
      port: 8080

But the CPU usage is still pretty high (~7%) so I opened an issue with cadvisor: https://github.com/google/cadvisor/issues/2523

I hope they have tips and I will keep this ticket updated if they have interesting conclusions!

aspro-at commented 4 years ago

I'm experiencing exactly the same.

frayos commented 4 years ago

Thank you @immortaly007 !

scrape_interval: 5s
scrape_interval: 15s

The above helped !

immortaly007 commented 4 years ago

Good to hear! I managed to bring the CPU usage down further by modifying the "housekeeping interval" and disabling statistics that weren't needed by the default (and my current) grafana dashboard setup.

To do this, under the cadvisor part of the docker-compose.yml file, add:

services:
#...
  cadvisor:
#  ...
    command:
      - '--docker_only' 
      - '--disable_metrics=disk,diskIO,tcp,udp,percpu,sched,process'
      - '--housekeeping_interval=10s'

aspro-at commented 4 years ago

Good to hear! I managed to bring the CPU usage down further by modifying the "housekeeping interval" and disabling statistics that weren't needed by the default (and my current) grafana dashboard setup.

To do this, under the cadvisor part of the docker-compose.yml file, add:
services:
#...
  cadvisor:
#  ...
    command:
      - '--docker_only' 
      - '--disable_metrics=disk,diskIO,tcp,udp,percpu,sched,process'
      - '--housekeeping_interval=10s'

The above helped! Down from 20 % to below 7 % cpu usage. Thanks.

github-actions[bot] commented 4 years ago

This issue has seen no activity and is marked as stale

balajeek commented 3 years ago

Good to hear! I managed to bring the CPU usage down further by modifying the "housekeeping interval" and disabling statistics that weren't needed by the default (and my current) grafana dashboard setup.

To do this, under the cadvisor part of the docker-compose.yml file, add:
services:
#...
  cadvisor:
#  ...
    command:
      - '--docker_only' 
      - '--disable_metrics=disk,diskIO,tcp,udp,percpu,sched,process'
      - '--housekeeping_interval=10s'

I recently setup cadvisor and prometheus and noticed 36% usage, when googled i ended up here, so i put in the command as you mentioned, after that cadvisor won't run, log shows a lot or lines, some that says error are:

-alsologtostderr log to standard error as well as files -logtostderr log to standard error instead of files -stderrthreshold value logs at or above this threshold go to stderr (default 2)

any idea?

vegasbrianc / prometheus