open-telemetry / opentelemetry-collector

OpenTelemetry Collector
https://opentelemetry.io
Apache License 2.0
4.12k stars 1.38k forks source link

Deadlock on startup when exporting is not possible #10518

Open martinsson opened 3 weeks ago

martinsson commented 3 weeks ago

Describe the bug The collector starts but does nothing on any of the pipelines All connection attempts are immediately aborted (http receiver)

From what I gather, this happens when an exporter (otlphttp) is unable to post data.

Perhaps this is not a bug, but I'm surprised that

Steps to reproduce Exceed metrics quota on grafana cloud, Include otlphttp exporter in one pipeline try to connect to post a log or connect to the http port telnet localhost 4317.

The connection is immediately closed

What did you expect to see? A log about the problem

That other pipelines work properly

What did you see instead? The whole app seems deadlocked

What version did you use? v1.1.0

What config did you use? approximately

receivers:
  otlp:
    protocols:
      grpc:
      http:

service:
  pipelines:
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [debug, otlphttp]

    traces:
      receivers: [otlp]
      processors: [ batch]
      exporters: [debug, otlp]

    logs:
      receivers: [otlp]
      processors: [batch]
      exporters: [debug, logging, loki]
martinsson commented 2 weeks ago

I've noted that the problem is also present for the otlp exporter, not only otlphttp. Furthermore it goes away systematically when the file_storage/otc extension is removed along with the corresponding sending_queue of the exporter

I'm sorry I didn't include it in the previous example.

I'll do some additional testing shortly to pinpoint the issue further