search

open-telemetry / opentelemetry-collector-contrib

Contrib repository for the OpenTelemetry Collector
https://opentelemetry.io
Apache License 2.0
2.88k stars 2.25k forks source link

prometheusexporter not attaching exemplars to metrics generated from spanmetricsconnector #28852

Closed ctrader2 closed 9 months ago

ctrader2 commented 10 months ago

Component(s)

connector/spanmetrics, exporter/prometheus

What happened?

Description

I am using the spanmetricsconnector in conjunction with prometheusexporter. I want to attach exemplars to the Prometheus metrics being generated. Based on the documentation for each of these components, I configured exemplars:enabled:true for the spanmetricsconnector and enable_open_metrics:true for the prometheusexporter.

It looks like exemplars are attached correctly to the OTEL metrics generated by spanmetricsconnector but are not carried forward/attached after conversion to Prometheus format.

Steps to Reproduce

  1. Run the collector with the specified configuration
  2. Send traces to the collector
  3. View the Prometheus metrics which are generated from the trace data using the spanmetricsconnector

Expected Result

Trace IDs attached to each Prometheus metric.

Actual Result

No trace information is attached.

Collector version

v0.88.0

Environment information

No response

OpenTelemetry Collector configuration

receivers:
  otlp:
    protocols:
      grpc:

processors:
  batch:
  filter/httponly:
    error_mode: ignore
    metrics:
      datapoint:
        - 'attributes["span.kind"] != "SPAN_KIND_SERVER" or attributes["http.status_code"] == nil' 

exporters:
  otlp:
    endpoint: grafana-agent:4317
    tls:
      insecure: true
  prometheus:
    endpoint: "0.0.0.0:8889"
    enable_open_metrics: true
    add_metric_suffixes: false
  logging:
    verbosity: detailed

connectors:
  spanmetrics:
    dimensions:
      - name: http.method
      - name: http.status_code
      - name: http.route
    exemplars:
      enabled: true

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [spanmetrics, otlp]
    metrics/spanmetrics:
      receivers: [spanmetrics]
      processors: [filter/httponly]
      exporters: [logging, prometheus]

Log output

Collector logs:

jobrunner-otel-collector_1      | 2023-11-01T19:16:29.040Z      info    MetricsExporter {"kind": "exporter", "data_type": "metrics", "name": "logging", "resource metrics": 1, "metrics": 2, "data points": 2}
jobrunner-otel-collector_1      | 2023-11-01T19:16:29.040Z      info    ResourceMetrics #0
jobrunner-otel-collector_1      | Resource SchemaURL:
jobrunner-otel-collector_1      | Resource attributes:
jobrunner-otel-collector_1      |      -> container.id: Str(3973966180a2a8394ff4d60c3e2806e7486725dae9b2595a64bc6b7db59effd9)
jobrunner-otel-collector_1      |      -> host.arch: Str(amd64)
jobrunner-otel-collector_1      |      -> host.name: Str(3973966180a2)
jobrunner-otel-collector_1      |      -> os.description: Str(Linux 5.15.123.1-microsoft-standard-WSL2)
jobrunner-otel-collector_1      |      -> os.type: Str(linux)
jobrunner-otel-collector_1      |      -> process.command_line: Str(/usr/lib/jvm/java-1.8-openjdk/jre/bin/java -javaagent:/opentelemetry-javaagent.jar app.jar --spring.profiles.active=localdev java -javaagent:/opentelemetry-javaagent.jar -jar /app.jar --spring.profiles.active=localdev)
jobrunner-otel-collector_1      |      -> process.executable.path: Str(/usr/lib/jvm/java-1.8-openjdk/jre/bin/java)
jobrunner-otel-collector_1      |      -> process.pid: Int(1)
jobrunner-otel-collector_1      |      -> process.runtime.description: Str(IcedTea OpenJDK 64-Bit Server VM 25.212-b04)
jobrunner-otel-collector_1      |      -> process.runtime.name: Str(OpenJDK Runtime Environment)
jobrunner-otel-collector_1      |      -> process.runtime.version: Str(1.8.0_212-b04)
jobrunner-otel-collector_1      |      -> service.name: Str(jobrunner)
jobrunner-otel-collector_1      |      -> telemetry.auto.version: Str(1.30.0)
jobrunner-otel-collector_1      |      -> telemetry.sdk.language: Str(java)
jobrunner-otel-collector_1      |      -> telemetry.sdk.name: Str(opentelemetry)
jobrunner-otel-collector_1      |      -> telemetry.sdk.version: Str(1.30.1)
jobrunner-otel-collector_1      | ScopeMetrics #0
jobrunner-otel-collector_1      | ScopeMetrics SchemaURL:
jobrunner-otel-collector_1      | InstrumentationScope spanmetricsconnector
jobrunner-otel-collector_1      | Metric #0
jobrunner-otel-collector_1      | Descriptor:
jobrunner-otel-collector_1      |      -> Name: calls
jobrunner-otel-collector_1      |      -> Description:
jobrunner-otel-collector_1      |      -> Unit:
jobrunner-otel-collector_1      |      -> DataType: Sum
jobrunner-otel-collector_1      |      -> IsMonotonic: true
jobrunner-otel-collector_1      |      -> AggregationTemporality: Cumulative
jobrunner-otel-collector_1      | NumberDataPoints #0
jobrunner-otel-collector_1      | Data point attributes:
jobrunner-otel-collector_1      |      -> service.name: Str(jobrunner)
jobrunner-otel-collector_1      |      -> span.name: Str(GET /monet/influx/version)
jobrunner-otel-collector_1      |      -> span.kind: Str(SPAN_KIND_SERVER)
jobrunner-otel-collector_1      |      -> status.code: Str(STATUS_CODE_UNSET)
jobrunner-otel-collector_1      |      -> http.method: Str(GET)
jobrunner-otel-collector_1      |      -> http.status_code: Int(200)
jobrunner-otel-collector_1      |      -> http.route: Str(/monet/influx/version)
jobrunner-otel-collector_1      | StartTimestamp: 2023-11-01 19:13:29.039592939 +0000 UTC
jobrunner-otel-collector_1      | Timestamp: 2023-11-01 19:16:29.040404546 +0000 UTC
jobrunner-otel-collector_1      | Value: 2
jobrunner-otel-collector_1      | Metric #1
jobrunner-otel-collector_1      | Descriptor:
jobrunner-otel-collector_1      |      -> Name: duration
jobrunner-otel-collector_1      |      -> Description:
jobrunner-otel-collector_1      |      -> Unit: ms
jobrunner-otel-collector_1      |      -> DataType: Histogram
jobrunner-otel-collector_1      |      -> AggregationTemporality: Cumulative
jobrunner-otel-collector_1      | HistogramDataPoints #0
jobrunner-otel-collector_1      | Data point attributes:
jobrunner-otel-collector_1      |      -> service.name: Str(jobrunner)
jobrunner-otel-collector_1      |      -> span.name: Str(GET /monet/influx/version)
jobrunner-otel-collector_1      |      -> span.kind: Str(SPAN_KIND_SERVER)
jobrunner-otel-collector_1      |      -> status.code: Str(STATUS_CODE_UNSET)
jobrunner-otel-collector_1      |      -> http.method: Str(GET)
jobrunner-otel-collector_1      |      -> http.status_code: Int(200)
jobrunner-otel-collector_1      |      -> http.route: Str(/monet/influx/version)
jobrunner-otel-collector_1      | StartTimestamp: 2023-11-01 19:13:29.039592939 +0000 UTC
jobrunner-otel-collector_1      | Timestamp: 2023-11-01 19:16:29.040409646 +0000 UTC
jobrunner-otel-collector_1      | Count: 2
jobrunner-otel-collector_1      | Sum: 1817.928041
jobrunner-otel-collector_1      | ExplicitBounds #0: 2.000000
jobrunner-otel-collector_1      | ExplicitBounds #1: 4.000000
jobrunner-otel-collector_1      | ExplicitBounds #2: 6.000000
jobrunner-otel-collector_1      | ExplicitBounds #3: 8.000000
jobrunner-otel-collector_1      | ExplicitBounds #4: 10.000000
jobrunner-otel-collector_1      | ExplicitBounds #5: 50.000000
jobrunner-otel-collector_1      | ExplicitBounds #6: 100.000000
jobrunner-otel-collector_1      | ExplicitBounds #7: 200.000000
jobrunner-otel-collector_1      | ExplicitBounds #8: 400.000000
jobrunner-otel-collector_1      | ExplicitBounds #9: 800.000000
jobrunner-otel-collector_1      | ExplicitBounds #10: 1000.000000
jobrunner-otel-collector_1      | ExplicitBounds #11: 1400.000000
jobrunner-otel-collector_1      | ExplicitBounds #12: 2000.000000
jobrunner-otel-collector_1      | ExplicitBounds #13: 5000.000000
jobrunner-otel-collector_1      | ExplicitBounds #14: 10000.000000
jobrunner-otel-collector_1      | ExplicitBounds #15: 15000.000000
jobrunner-otel-collector_1      | Buckets #0, Count: 0
jobrunner-otel-collector_1      | Buckets #1, Count: 0
jobrunner-otel-collector_1      | Buckets #2, Count: 0
jobrunner-otel-collector_1      | Buckets #3, Count: 0
jobrunner-otel-collector_1      | Buckets #4, Count: 0
jobrunner-otel-collector_1      | Buckets #5, Count: 0
jobrunner-otel-collector_1      | Buckets #6, Count: 0
jobrunner-otel-collector_1      | Buckets #7, Count: 0
jobrunner-otel-collector_1      | Buckets #8, Count: 1
jobrunner-otel-collector_1      | Buckets #9, Count: 0
jobrunner-otel-collector_1      | Buckets #10, Count: 0
jobrunner-otel-collector_1      | Buckets #11, Count: 0
jobrunner-otel-collector_1      | Buckets #12, Count: 1
jobrunner-otel-collector_1      | Buckets #13, Count: 0
jobrunner-otel-collector_1      | Buckets #14, Count: 0
jobrunner-otel-collector_1      | Buckets #15, Count: 0
jobrunner-otel-collector_1      | Buckets #16, Count: 0
jobrunner-otel-collector_1      | Exemplars:
jobrunner-otel-collector_1      | Exemplar #0
jobrunner-otel-collector_1      |      -> Trace ID: 249460d4c98040997d836f00a190c7da
jobrunner-otel-collector_1      |      -> Span ID: 20f8de9b97fbe1d8
jobrunner-otel-collector_1      |      -> Timestamp: 2023-11-01 19:16:29.040409646 +0000 UTC
jobrunner-otel-collector_1      |      -> Value: 311.627870
jobrunner-otel-collector_1      |       {"kind": "exporter", "data_type": "metrics", "name": "logging"}

Response from scrape endpoint:

# HELP calls 
# TYPE calls counter
calls{http_method="GET",http_route="/monet/influx/version",http_status_code="200",job="jobrunner",service_name="jobrunner",span_kind="SPAN_KIND_SERVER",span_name="GET /monet/influx/version",status_code="STATUS_CODE_UNSET"} 6
# HELP duration 
# TYPE duration histogram
duration_bucket{http_method="GET",http_route="/monet/influx/version",http_status_code="200",job="jobrunner",service_name="jobrunner",span_kind="SPAN_KIND_SERVER",span_name="GET /monet/influx/version",status_code="STATUS_CODE_UNSET",le="2"} 0
duration_bucket{http_method="GET",http_route="/monet/influx/version",http_status_code="200",job="jobrunner",service_name="jobrunner",span_kind="SPAN_KIND_SERVER",span_name="GET /monet/influx/version",status_code="STATUS_CODE_UNSET",le="4"} 0
duration_bucket{http_method="GET",http_route="/monet/influx/version",http_status_code="200",job="jobrunner",service_name="jobrunner",span_kind="SPAN_KIND_SERVER",span_name="GET /monet/influx/version",status_code="STATUS_CODE_UNSET",le="6"} 0
duration_bucket{http_method="GET",http_route="/monet/influx/version",http_status_code="200",job="jobrunner",service_name="jobrunner",span_kind="SPAN_KIND_SERVER",span_name="GET /monet/influx/version",status_code="STATUS_CODE_UNSET",le="8"} 0
duration_bucket{http_method="GET",http_route="/monet/influx/version",http_status_code="200",job="jobrunner",service_name="jobrunner",span_kind="SPAN_KIND_SERVER",span_name="GET /monet/influx/version",status_code="STATUS_CODE_UNSET",le="10"} 0
duration_bucket{http_method="GET",http_route="/monet/influx/version",http_status_code="200",job="jobrunner",service_name="jobrunner",span_kind="SPAN_KIND_SERVER",span_name="GET /monet/influx/version",status_code="STATUS_CODE_UNSET",le="50"} 0
duration_bucket{http_method="GET",http_route="/monet/influx/version",http_status_code="200",job="jobrunner",service_name="jobrunner",span_kind="SPAN_KIND_SERVER",span_name="GET /monet/influx/version",status_code="STATUS_CODE_UNSET",le="100"} 0
duration_bucket{http_method="GET",http_route="/monet/influx/version",http_status_code="200",job="jobrunner",service_name="jobrunner",span_kind="SPAN_KIND_SERVER",span_name="GET /monet/influx/version",status_code="STATUS_CODE_UNSET",le="200"} 0
duration_bucket{http_method="GET",http_route="/monet/influx/version",http_status_code="200",job="jobrunner",service_name="jobrunner",span_kind="SPAN_KIND_SERVER",span_name="GET /monet/influx/version",status_code="STATUS_CODE_UNSET",le="400"} 5
duration_bucket{http_method="GET",http_route="/monet/influx/version",http_status_code="200",job="jobrunner",service_name="jobrunner",span_kind="SPAN_KIND_SERVER",span_name="GET /monet/influx/version",status_code="STATUS_CODE_UNSET",le="800"} 5
duration_bucket{http_method="GET",http_route="/monet/influx/version",http_status_code="200",job="jobrunner",service_name="jobrunner",span_kind="SPAN_KIND_SERVER",span_name="GET /monet/influx/version",status_code="STATUS_CODE_UNSET",le="1000"} 5
duration_bucket{http_method="GET",http_route="/monet/influx/version",http_status_code="200",job="jobrunner",service_name="jobrunner",span_kind="SPAN_KIND_SERVER",span_name="GET /monet/influx/version",status_code="STATUS_CODE_UNSET",le="1400"} 5
duration_bucket{http_method="GET",http_route="/monet/influx/version",http_status_code="200",job="jobrunner",service_name="jobrunner",span_kind="SPAN_KIND_SERVER",span_name="GET /monet/influx/version",status_code="STATUS_CODE_UNSET",le="2000"} 6
duration_bucket{http_method="GET",http_route="/monet/influx/version",http_status_code="200",job="jobrunner",service_name="jobrunner",span_kind="SPAN_KIND_SERVER",span_name="GET /monet/influx/version",status_code="STATUS_CODE_UNSET",le="5000"} 6
duration_bucket{http_method="GET",http_route="/monet/influx/version",http_status_code="200",job="jobrunner",service_name="jobrunner",span_kind="SPAN_KIND_SERVER",span_name="GET /monet/influx/version",status_code="STATUS_CODE_UNSET",le="10000"} 6
duration_bucket{http_method="GET",http_route="/monet/influx/version",http_status_code="200",job="jobrunner",service_name="jobrunner",span_kind="SPAN_KIND_SERVER",span_name="GET /monet/influx/version",status_code="STATUS_CODE_UNSET",le="15000"} 6
duration_bucket{http_method="GET",http_route="/monet/influx/version",http_status_code="200",job="jobrunner",service_name="jobrunner",span_kind="SPAN_KIND_SERVER",span_name="GET /monet/influx/version",status_code="STATUS_CODE_UNSET",le="+Inf"} 6
duration_sum{http_method="GET",http_route="/monet/influx/version",http_status_code="200",job="jobrunner",service_name="jobrunner",span_kind="SPAN_KIND_SERVER",span_name="GET /monet/influx/version",status_code="STATUS_CODE_UNSET"} 3022.686396
duration_count{http_method="GET",http_route="/monet/influx/version",http_status_code="200",job="jobrunner",service_name="jobrunner",span_kind="SPAN_KIND_SERVER",span_name="GET /monet/influx/version",status_code="STATUS_CODE_UNSET"} 6

Additional context

No response

github-actions[bot] commented 10 months ago

Pinging code owners:

See Adding Labels via Comments if you do not have permissions to add labels yourself.

yantingqiu commented 10 months ago

I would suggest giving the following configuration a try.

exporters:
  otlp:
    endpoint: grafana-agent:4317
    tls:
      insecure: true
  prometheus:
    endpoint: "0.0.0.0:8889"
    enable_open_metrics: true
    add_metric_suffixes: false
    resource_to_telemetry_conversion: true   //  If `enabled` is `true`, all the resource attributes will be converted to metric labels by default.
crobert-1 commented 10 months ago

@ctrader2 have you been able to test @yantingqiu's shared configuration?

Note: I believe the correct configuration for this option would be:

resource_to_telemetry_conversion:
  enabled: true
crobert-1 commented 9 months ago

I'm going to close for now as I believe the suggested configuration update will resolve the issue. Please let us know though if you get a chance to test and that's not the case!