Exporter metrics (otelcol_exporter_queue_capacity, otelcol_exporter_queue_size) are generated only for a single exporter on otel collector 0.96.0

tqi-raurora commented 4 months ago

Component(s)

No response

What happened?

Description

On version 0.96.0, when setting 2 different exporters, then querying localhost:8888/metrics, only one exporter is shown on the metrics otelcol_exporter_queue_size and otelcol_exporter_queue_capacity. The exporter that is shown appears to be random, and can change if the service is restarted. On version 0.92.0 this was working as expected, with metrics for both exporters being shown.

Steps to Reproduce

Create a configuration with 2 exporters (example: 2 otlp exporters) Query local telemetry endpoint at localhost:8888/metrics

Expected Result

Both endpoints should be shown. Example:

tqi_raurora@ni-26829-4p:~$ otelcol-contrib --version
otelcol-contrib version 0.92.0
tqi_raurora@ni-26829-4p:~$ curl -s 0:8888/metrics | grep ^otelcol_exporter | grep queue
otelcol_exporter_queue_capacity{exporter="otlp/jaeger-1",service_instance_id="721c4da6-9e52-4119-99e5-148e88e1dba6",service_name="otelcol-contrib",service_version="0.92.0"} 1000
otelcol_exporter_queue_capacity{exporter="otlp/jaeger-2",service_instance_id="721c4da6-9e52-4119-99e5-148e88e1dba6",service_name="otelcol-contrib",service_version="0.92.0"} 1000
otelcol_exporter_queue_size{exporter="otlp/jaeger-1",service_instance_id="721c4da6-9e52-4119-99e5-148e88e1dba6",service_name="otelcol-contrib",service_version="0.92.0"} 0
otelcol_exporter_queue_size{exporter="otlp/jaeger-2",service_instance_id="721c4da6-9e52-4119-99e5-148e88e1dba6",service_name="otelcol-contrib",service_version="0.92.0"} 0

Actual Result

Only metrics for one of the exporter is shown:

tqi_raurora@ni-26829-4p:~$ otelcol-contrib --version
otelcol-contrib version 0.96.0
tqi_raurora@ni-26829-4p:~$ curl -s 0:8888/metrics | grep ^otelcol_exporter | grep queue
otelcol_exporter_queue_capacity{exporter="otlp/jaeger-1",service_instance_id="c9c9dd6a-a367-42d1-a7c7-633c47b51a0a",service_name="otelcol-contrib",service_version="0.96.0"} 1000
otelcol_exporter_queue_size{exporter="otlp/jaeger-1",service_instance_id="c9c9dd6a-a367-42d1-a7c7-633c47b51a0a",service_name="otelcol-contrib",service_version="0.96.0"} 0

Collector version

0.96.0

Environment information

No response

OpenTelemetry Collector configuration

receivers:
  otlp/receiver:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

exporters:
  otlp/jaeger-1:
    endpoint: 0.0.0.0:14317
    tls:
      insecure: true
  otlp/jaeger-2:
    endpoint: 0.0.0.0:14317
    tls:
      insecure: true

service:
  pipelines:

    traces:
      receivers: [otlp/receiver]
      processors: []
      exporters: [otlp/jaeger-1, otlp/jaeger-2]

Log output

No response

Additional context

I had opened the original issue on the contrib repository: https://github.com/open-telemetry/opentelemetry-collector-contrib/issues/31645 But @crobert-1 determined it's an issue on the core repository, so I'm opening it here

TylerHelmuth commented 4 months ago

@tqi-raurora are both exporters taking traffic at the same time?

tqi-raurora commented 4 months ago

@TylerHelmuth Yes In my example I have a single otlp receiver, sending data to two otlp exporters.

But from what I tested it happens whenever theres more than one exporter (on our environment we got spans being sent to jaeger and metrics being sent to load balancer exporter, wich is a problem because we end up only with metrics for otlp exporter or load balancer exporter)

It seems whatever exporter gets to expose the metric is random, here I restarted the service without changing config.yaml and the exporter changed from otlp/jaeger-1 to otlp/jaeger-2:

tqi_raurora@ni-26829-4p:~$ curl -s 0:8888/metrics | grep ^otelcol_exporter | grep queue
otelcol_exporter_queue_capacity{exporter="otlp/jaeger-1",service_instance_id="e42887e0-4df6-4f7f-8ec6-9f5207636adc",service_name="otelcol-contrib",service_version="0.96.0"} 1000
otelcol_exporter_queue_size{exporter="otlp/jaeger-1",service_instance_id="e42887e0-4df6-4f7f-8ec6-9f5207636adc",service_name="otelcol-contrib",service_version="0.96.0"} 0

tqi_raurora@ni-26829-4p:~$ systemctl restart otelcol-contrib.service 

tqi_raurora@ni-26829-4p:~$ curl -s 0:8888/metrics | grep ^otelcol_exporter | grep queue
otelcol_exporter_queue_capacity{exporter="otlp/jaeger-2",service_instance_id="c2fb111a-e929-4350-82af-bc4049f8f6f2",service_name="otelcol-contrib",service_version="0.96.0"} 1000
otelcol_exporter_queue_size{exporter="otlp/jaeger-2",service_instance_id="c2fb111a-e929-4350-82af-bc4049f8f6f2",service_name="otelcol-contrib",service_version="0.96.0"} 0

TylerHelmuth commented 4 months ago

@dmitryax could this be related to any recent exporterhelper work?

open-telemetry / opentelemetry-collector