[receiver/dockerstatsreceiver] Exported metrics are not labeled with configured labels

[hashemmm96]

Describe the bug dockerstatsreceiver doesn't add labels from container_labels_to_metric_labels and env_vars_to_metric_labels config options to exported metrics.

Steps to reproduce

1. Modify otelcontribcol docker image (cmd/otelcontribcol/Dockerfile) to use USER_UID=0.
2. make docker-otelcontribcol
3. export OTELCOL_IMG=otelcontribcol
4. Run docker-compose in examples/demo but with modified otel-collector-config.yaml and docker-compose.yaml below:

# docker-compose.yaml
version: "2"
services:

  # Jaeger
  jaeger-all-in-one:
    image: jaegertracing/all-in-one:latest
    ports:
      - "16686:16686"
      - "14268"
      - "14250"

  # Zipkin
  zipkin-all-in-one:
    image: openzipkin/zipkin:latest
    ports:
      - "9411:9411"

  # Collector
  otel-collector:
    image: ${OTELCOL_IMG}
    command: ["--config=/etc/otel-collector-config.yaml", "${OTELCOL_ARGS}"]
    volumes:
      - ./otel-collector-config.yaml:/etc/otel-collector-config.yaml
      - /var/run/docker.sock:/var/run/docker.sock
    ports:
      - "1888:1888"   # pprof extension
      - "8888:8888"   # Prometheus metrics exposed by the collector
      - "8889:8889"   # Prometheus exporter metrics
      - "13133:13133" # health_check extension
      - "4317"        # OTLP gRPC receiver
      - "55670:55679" # zpages extension
    depends_on:
      - jaeger-all-in-one
      - zipkin-all-in-one

  demo-client:
    build:
      dockerfile: Dockerfile
      context: ./client
    environment:
      - OTEL_EXPORTER_OTLP_ENDPOINT=otel-collector:4317
      - DEMO_SERVER_ENDPOINT=http://demo-server:7080/hello
    depends_on:
      - demo-server

  demo-server:
    build:
      dockerfile: Dockerfile
      context: ./server
    environment:
      - OTEL_EXPORTER_OTLP_ENDPOINT=otel-collector:4317
    ports:
      - "7080"
    depends_on:
      - otel-collector

  prometheus:
    container_name: prometheus
    image: prom/prometheus:latest
    volumes:
      - ./prometheus.yaml:/etc/prometheus/prometheus.yml
    ports:
      - "9090:9090"

# otel-collector-config.yaml
receivers:
  otlp:
    protocols:
      grpc:

  docker_stats:
    collection_interval: 5s
    api_version: 1.41
    container_labels_to_metric_labels:
      com.docker.compose.service: service
    env_vars_to_metric_labels:
      DEMO_SERVER_ENDPOINT: demo-server-endpoint

exporters:
  prometheus:
    endpoint: "0.0.0.0:8889"
    const_labels:
      label1: value1

  logging:

  zipkin:
    endpoint: "http://zipkin-all-in-one:9411/api/v2/spans"
    format: proto

  jaeger:
    endpoint: jaeger-all-in-one:14250
    tls:
      insecure: true

processors:
  batch:

extensions:
  health_check:
  pprof:
    endpoint: :1888
  zpages:
    endpoint: :55679

service:
  extensions: [pprof, zpages, health_check]
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [logging, zipkin, jaeger]
    metrics:
      receivers: [otlp, docker_stats]
      processors: [batch]
      exporters: [logging, prometheus]

5. Check prometheus at http://0.0.0.0:9090. Check any container_* metric, e.g. container_cpu_percent

What did you expect to see? One series for each container in the docker-compose env should be visible in prometheus, i.e. five graphs. Each series should be labeled with the docker-compose service name and the demo-client service should have an additional label from the DEMO_SERVER_ENDPOINT env var.

What did you see instead? One series in total, with no labels from the otel-collector-config.yaml showing.

What version did you use? commit 32e14e6

What config did you use? see above

Environment OS: Ubuntu 21.10

Additional context

[hashemmm96]

Update: Suggested fix here https://github.com/open-telemetry/opentelemetry-collector-contrib/pull/8543

[pmm-sumo]

As discussed in the PR, I think the proposed change is not the right solution to the problem. And the underlying cause is not really a bug but rather a misconfiguration (though to be honest, it's very easy to miss that). dockerstats receiver already puts the attributes at the resource level.

I don't know prometheus exporter very well, but I think by default it does not include resource-level attributes. To change it, resource_to_telemetry_conversion enabled flag needs to be set true.

So I think all that needs to be done is following change to otel-collector-config.yaml:

exporters:
  prometheus:
    endpoint: "0.0.0.0:8889"
    resource_to_telemetry_conversion: 
      enabled: true
    const_labels:
      label1: value1

[hashemmm96]

Thanks a lot for the help, there was much confusion on my part. I will close the PR as well.