search

stefanprodan / dockprom

Docker hosts and containers monitoring with Prometheus, Grafana, cAdvisor, NodeExporter and AlertManager
MIT License
5.97k stars 1.71k forks source link

Cadvisor high cpu fix #288

Open razem-io opened 11 months ago

razem-io commented 11 months ago

If I am not mistaken, those settings won't affect the current dashboards at all, while reducing CPU usage of cadvisor by a lot. On my test system, the average CPU usage was reduced from ~7% to ~2%.

All the credit goes to this comment: https://github.com/google/cadvisor/issues/2523#issuecomment-785094428

image

As a nice side effect, it also seems to reduce the memory needed.

image

kroese commented 10 months ago

@razem-io You forgot the most important setting discovered in that issue:

store_container_labels=false

Using that one has the biggest impact on CPU usage for me, much more than changing the scrape interval like you did now.

razem-io commented 10 months ago

@kroese funny enough that did not change anything for me. I guess it depends on the individual setup. Do you know what the impact is? Not sure if the labels are used somewhere in the dashboard. I can add it, if does not do much.

kroese commented 10 months ago

@razem-io It does not change anything in the dashboards, so it wouldn't hurt to use this setting as dock-prom is not using the labels that it generates.

Strange that it didnt make any difference in your case. Maybe it depends on the amount of containers for example (I have 70+ running). Or the fact that I collect the data from 3 different remote docker machines over the network, not just from a local instance. But I could see a massive difference in CPU usage after disabling the labels.

patrontheo commented 5 months ago

+1 @razem-io, your solution also helped my to decrease cadvisor CPU usage, thank you!

image

I haven't tried @kroese solution yet

razem-io commented 5 months ago

@kroese I added your suggestion to the PR. Sorry, I totally forgot about it.

patrontheo commented 5 months ago

@razem-io unfortunately, adding the - '-store_container_labels=false' will prevent prometheus to have access to the label 'monitoring' that is used in the monitoring containers. Then, the dashboard 'docker containers' will also contains the monitoring containers (which it shouldn't), and the dashboard 'monitor services' will be more or less empty. @kroese Do you think it is possible to filter the labels that are stored (to keep only container_label_org_label_schema_group and not all the others) ?

razem-io commented 5 months ago

@patrontheo thanks! I reverted the commit. Haven't tested it myself yet. Should have done that before pushing the changes.