search

open-telemetry / opentelemetry-erlang-contrib

OpenTelemetry instrumentation for Erlang & Elixir
https://opentelemetry.io
Apache License 2.0
162 stars 115 forks source link

No way to set max compression size limit #159

Open sashaafm opened 1 year ago

sashaafm commented 1 year ago

Describe the bug Maybe not really a bug but more a lack of configuration or adherence to the spec.

We're seeing this kind of error:

OTLP grpc export failed with GRPC status 8 : grpc: received message after decompression larger than max (13104533 vs. 4194304)

Which looks like the message is too large and hits the 4194304 size limit. However, some people like in this issue here seem to have configurable ways of increasing this limit.

Possibly a workaround is to play with these env vars decreasing their scheduled delay and max queue size to deliver messages quicker before they reach this size limit?

Expected behavior A way to avoid this error, either by a new configuration or some documentation explaining how to tackle it.

Additional context

tsloughter commented 1 year ago

@sashaafm you are seeing this error in tempo?

sashaafm commented 1 year ago

No, this happens in the Elixir console. We’re using Datadog as a metrics aggregator/visualizer. However, this happens in the instance where the app is running (alongside the Datadog Agent).

tsloughter commented 1 year ago

Adding here what I said in Slack so it isn't lost in time :)

We don't yet support OTEL_BSP_MAX_EXPORT_BATCH_SIZE but do have OTEL_BSP_SCHEDULE_DELAY_MILLIS (bsp_scheduled_delay_ms in the application configuration) which defaults to 5 seconds.