search

google / cadvisor

Analyzes resource usage and performance characteristics of running containers.
Other
16.92k stars 2.31k forks source link

new feature: labels mapping exported in metrics #1447

Open muradm opened 8 years ago

muradm commented 8 years ago

As per discussion in #1426 here, proposal with configurable set of labels seems to be most reasonable. In addition to the list it could be a map, something like:

[-container.labels <label_name_as_set_on_container[:label_name_as_should_be_exported]> ... ]

where label_name_as_should_be_exported is optional, if it is needed to change the name of exported label.

dashpole commented 7 years ago

Feel free to open a PR for this, and Ill take a look

SamSaffron commented 5 years ago

Note @dashpole on one of my servers I am seeing a payload of 27.3M of payload mostly due to metrics looking like this:

container_cpu_load_average_10s{container_label_build_date="",container_label_license="",container_label_maintainer="",container_label_name="",container_label_org_discourse_puppet_command_hash="",container_label_org_discourse_service_1514="",container_label_org_discourse_service_5044="",container_label_org_discourse_service_5150="",container_label_org_discourse_service_5151="",container_label_org_discourse_service_5432="",container_label_org_discourse_service_6432="",container_label_org_discourse_service_8053="",container_label_org_discourse_service_8080="",container_label_org_discourse_service_8080_tags="",container_label_org_discourse_service_9113="",container_label_org_discourse_service_9113_tags="",container_label_org_discourse_service_9127="",container_label_org_discourse_service_9127_tags="",container_label_org_discourse_service_9304="",container_label_org_discourse_service_9304_tags="",container_label_org_discourse_service_9600="",container_label_org_discourse_service_9696="",container_label_org_discourse_service_9696_tags="",container_label_org_discourse_service_9797="",container_label_org_discourse_service_9797_tags="",container_label_org_discourse_service__logstash_beats_instance="",container_label_org_discourse_service__logstash_beats_port="",container_label_org_discourse_service__logstash_stats_instance="",container_label_org_discourse_service__logstash_stats_port="",container_label_org_discourse_service__prom_exp_instance="",container_label_org_discourse_service__prom_exp_port="",container_label_restartcount="",container_label_vendor="",id="/",image="",name=""} 0

Profiling shows that about 27% of our CPU time is in labelPairsToText ... our cadvisor is pegged at 60% of one core ... if we disable collection of metrics from prometheus it goes down to 6% of one core. This machine has 180 containers

SamSaffron commented 5 years ago

I think a very very important note to anyone coming across this ticket @dashpole is that now we have:

-store_container_labels=false

This is the graphical impact of what kind of a change this made to one of our servers.

5690a258450ef67ed21d410b66f4b066f75f1821

It also halved CPU usage on prometheus.

I think it is probably correct to make a breaking change here for the greater good and defaulting store_container_labels to false.

dashpole commented 5 years ago

@SamSaffron That is quite a change. I agree that we should change the default. Can you open a PR to do this?