Exporting failed, dropping data because of high memory usage

[Akrammousselmal]

Describe the bug

I keep getting error from my opentelemetry-agent of data dropped due to high memory usage. This is happening even though the curr_mem_mib is only at 24 mib..

Steps to reproduce

Use opentelemetry-agent:0.98.0 with otlp exporter

What did you expect to see?

Traces being sent..

What did you see instead?

Errors Mostly this: error exporterhelper/queue_sender.go:101 Exporting failed. Dropping data. {"kind": "exporter", "data_type": "traces", "name": "otlp", "error": "not retryable error: Permanent error: rpc error: code = Unknown desc = data drop │ │ ped due to high memory usage; data dropped due to high memory usage"

And this

Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "traces", "name": "otlp", "error": "rpc error: code = DeadlineExceeded desc = context deadlin │ │ e exceeded", "interval": "4.908240855s"}

What version did you use?

v0.98.0 What config did you use?

  receivers:
      jaeger:
        protocols:
          grpc:
            endpoint: "127.0.0.1:14249"
  otlp:
    protocols:
      grpc:
        endpoint: "127.0.0.1:14248"

exporters:
  otlp:
    endpoint: XXXXXX:15251
    tls:
      insecure: true
    sending_queue:
      num_consumers: 4
      queue_size: 100
    retry_on_failure:
      enabled: true

processors:
  memory_limiter:
    # 80% of maximum memory up to 2G
    limit_mib: 400
    # 25% of limit up to 2G
    spike_limit_mib: 100
    check_interval: 5s

  probabilistic_sampler:
    hash_seed: 22
    sampling_percentage: 100

extensions:
  health_check: {}

service:
  telemetry:
    logs:
      level: "debug"
  extensions:
    - health_check
  pipelines:
    traces:
      receivers:
        - jaeger
        - otlp
      processors:
        - probabilistic_sampler
        - memory_limiter
        - attributes/insert
      exporters:
        - otlp

Environment

Additional context

Here the errors:

[Akrammousselmal]

Using version 0.90.0 is better since I get full exporting working but still have many that fails

[TylerHelmuth]

@Akrammousselmal I believe those exporter errors are all coming from whatever is receiving your data.