github-actions[bot]
commented
1 year ago Pinging code owners:
- receiver/dockerstats: @rmfitzpatrick @jamesmoessis
See Adding Labels via Comments if you do not have permissions to add labels yourself.
Closed toxic0berliner closed 1 year ago
github-actions[bot]
commented
1 year ago Pinging code owners:
See Adding Labels via Comments if you do not have permissions to add labels yourself.
jamesmoessis
commented
1 year ago Hi @toxic0berliner the container information is part of the resource attributes, which is at a "higher level". I'm not sure how the prom exporter converts that or where it displays that information.
I'd be interested to see the output of a detailed loglevel in the logging exporter. That would tell me the exact metrics that are being exported, and not manipulated by the prom exporter.
Can you post the output with the following config?
exporters:
logging:
verbosity: detailed
toxic0berliner
commented
1 year ago wow, got some new logs with this !!!
This goes on for quite a whille like this with pretty much everything I'd like to see in prom indeed ;)
But still nothing more to see in /metrics...
Just to be fully transparent, here is the real original config file with all the comments and messyness, just removed the batch processor just in case, turns out it doesn't change anything...
As I'm seeing all these stats I believe it answers one of my question as docker_stats should indeed exports metrics for each container, and otel-collector with docker-stats is a good candidate for me to replace cadvisor that is eating up my CPU on my poor synology NAS
Seeing it shows up in the logs now I'm guessing I messed up some config, still very new to otel, sorry if I flagged this as a docke_stats issue where it might very well be a misstake by me, if you can help me find it that would be great !
Not that I tried the same setup on an ubuntu VM runing docker, just to insure there is no trickery specifi to synology and no, same setup also does not give the per container metrics in the /metrics endpoint of otel-collector :(
jamesmoessis
commented
1 year ago No problem at all @toxic0berliner - it does looks like it's something to do with the prom exporter. My advice would be to look at how the prom exporter handles resource attributes. Unfortunately on the top of my head I don't have the answer to why those attributes aren't showing up on /metrics.
toxic0berliner
commented
1 year ago Would you maybe know how to enable some more logs on the prom exporter ?
github-actions[bot]
commented
1 year ago This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.
Pinging code owners:
See Adding Labels via Comments if you do not have permissions to add labels yourself.
toxic0berliner
commented
1 year ago Are we sure here it's not an issue with this collector and with the Prometheus exporter instead?
toxic0berliner
commented
1 year ago Can someone maybe relabel this as exporter/prometheus and help me have a look? Can't seem to find anything to solve it by myself :(
Artyomlad
commented
1 year ago Same here. To verify we used this receiver with "file exporter" (instead of Prometheus) and indeed file content contained all the docker container metrics.
In Prometheus exporter we used the flag "resource_to_telemetry_conversion: true" and it got different output that do include the individual docker containers.
toxic0berliner
commented
1 year ago Amazing will test that out thanks for the hint on this hidden setting
toxic0berliner
commented
1 year ago got it working, thanks a lot, now need to submit an enhancement request tu expose the labels as I am missing container_label_com_docker_compose_project at least, very much needed to group metrics by compose projects ;)
toxic0berliner
commented
1 year ago Can close it, works like a charm an config can be adjusted to expose docker labels so all fine and working, thanks again @Artyomlad for the kind help!
Component(s)
receiver/dockerstats
Describe the issue you're reporting
I'm not recieving any stats on individual containers.
I started otel-collector using following compose settings :
docker-compose.yml
``` version: "3.9" services: otel-collector: image: otel/opentelemetry-collector-contrib hostname: otel-collector container_name: otel-collector command: ["--config=/etc/otel-collector-config.yml"] user: "0" # needed to access docker.sock labels: traefik.http.services.otel-collector.loadbalancer.server.port: 8889 traefik.http.routers.otel-collector.service: otel-collector volumes: - ./otel-collector-config.yml:/etc/otel-collector-config.yml - /var/run/docker.sock:/var/run/docker.sock:rw - /var/lib/docker/containers:/var/lib/docker/containers:ro ports: # - 1888:1888 # pprof extension # - 8888:8888 # Prometheus metrics exposed by the collector - 8889:8889 # Prometheus exporter metrics # - 13133:13133 # health_check extension # - 4317:4317 # OTLP gRPC receiver # - 4318:4318 # OTLP http receiver # - 55679:55679 # zpages extension ```And this is my
otel-collector-config.yml:otel-collector-config.yml
``` receivers: otlp: protocols: grpc: docker_stats: endpoint: "unix:///var/run/docker.sock" api_version: 1.41 collection_interval: 2s timeout: 20s env_vars_to_metric_labels: - com.docker.compose.project: container_stack_name - com.docker.compose.service: container_service_name exporters: prometheus: endpoint: "0.0.0.0:8889" logging: processors: batch: extensions: health_check: pprof: endpoint: :1888 zpages: endpoint: :55679 service: telemetry: logs: level: "debug" extensions: [pprof, zpages, health_check] pipelines: metrics: receivers: [docker_stats] processors: [] exporters: [logging, prometheus] ```All I'm getting under
http://myhost:8080/metricsis this:/metrics
``` # HELP container_cpu_percent Percent of CPU used by the container. # TYPE container_cpu_percent gauge container_cpu_percent 0.4280678815489749 # HELP container_cpu_usage_kernelmode Time spent by tasks of the cgroup in kernel mode (Linux). Time spent by all container processes in kernel mode (Windows). # TYPE container_cpu_usage_kernelmode counter container_cpu_usage_kernelmode 2.4254e+11 # HELP container_cpu_usage_total Total CPU time consumed. # TYPE container_cpu_usage_total counter container_cpu_usage_total 6.95768776915e+11 # HELP container_cpu_usage_usermode Time spent by tasks of the cgroup in user mode (Linux). Time spent by all container processes in user mode (Windows). # TYPE container_cpu_usage_usermode counter container_cpu_usage_usermode 4.2274e+11 # HELP container_memory_percent Percentage of memory used. # TYPE container_memory_percent gauge container_memory_percent 0.2483825322600396 # HELP container_memory_total_cache Total amount of memory used by the processes of this cgroup (and descendants) that can be associated with a block on a block device. Also accounts for memory used by tmpfs. # TYPE container_memory_total_cache gauge container_memory_total_cache 5.5816192e+07 # HELP container_memory_usage_limit Memory limit of the container. # TYPE container_memory_usage_limit gauge container_memory_usage_limit 1.6624267264e+10 # HELP container_memory_usage_total Memory usage of the container. This excludes the total cache. # TYPE container_memory_usage_total gauge container_memory_usage_total 4.1291776e+07 # HELP container_network_io_usage_rx_bytes Bytes received by the container. # TYPE container_network_io_usage_rx_bytes counter container_network_io_usage_rx_bytes{interface="eth0"} 3.728707676e+09 container_network_io_usage_rx_bytes{interface="eth1"} 8.354612e+06 # HELP container_network_io_usage_rx_dropped Incoming packets dropped. # TYPE container_network_io_usage_rx_dropped counter container_network_io_usage_rx_dropped{interface="eth0"} 0 container_network_io_usage_rx_dropped{interface="eth1"} 0 # HELP container_network_io_usage_tx_bytes Bytes sent. # TYPE container_network_io_usage_tx_bytes counter container_network_io_usage_tx_bytes{interface="eth0"} 3.734452545e+09 container_network_io_usage_tx_bytes{interface="eth1"} 1.1080344e+07 # HELP container_network_io_usage_tx_dropped Outgoing packets dropped. # TYPE container_network_io_usage_tx_dropped counter container_network_io_usage_tx_dropped{interface="eth0"} 0 container_network_io_usage_tx_dropped{interface="eth1"} 0 ```As such, I'm not seeing each container's CPU usage and so on...
I do see in the otel-collector container logs that it's supposed to be fetching them :
otel-collector logs
``` otel-collector | 2023-04-28T13:55:41.186Z debug prometheusexporter@v0.75.0/accumulator.go:90 accumulating metric: container.memory.usage.limit {"kind": "exporter", "data_type": "metrics", " name": "prometheus"} otel-collector | 2023-04-28T13:55:41.186Z debug prometheusexporter@v0.75.0/accumulator.go:90 accumulating metric: container.memory.usage.total {"kind": "exporter", "data_type": "metrics", " name": "prometheus"} otel-collector | 2023-04-28T13:55:41.186Z debug docker@v0.75.0/docker.go:162 Fetching container stats. {"kind": "receiver", "name": "docker_stats", "data_type": "metrics", "id": "8dbb3870e5 8153a523c2a83f48561b3a8cf11d55a07c93520b4bea5f5a96a3ca"} otel-collector | 2023-04-28T13:55:41.186Z debug docker@v0.75.0/docker.go:162 Fetching container stats. {"kind": "receiver", "name": "docker_stats", "data_type": "metrics", "id": "b94f3e8a1e 3acaf725f6bdd875b622e2433249aa15e86c909e55fb26f02177bc"} ```I think I've solved the access issues with
user: "0"since I'm not seeing anyPermission deniedanymore...Can someone help ? Or am I expecting to see stats for each container where this collector only gets overall system stats ?