search

open-telemetry / opentelemetry-java-instrumentation

OpenTelemetry auto-instrumentation and instrumentation libraries for Java
https://opentelemetry.io
Apache License 2.0
1.94k stars 846 forks source link

Failed to export spans. Server responded with gRPC status code 8. Error message: RATE_LIMITED #12449

Closed 1095071913 closed 1 week ago

1095071913 commented 1 week ago

Describe the bug

[otel.javaagent 2024-10-16 17:28:52:560 +0800] [OkHttp http://xxx:4317/...] WARN io.opentelemetry.exporter.internal.grpc.GrpcExporter - Failed to export spans. Server responded with gRPC status code 8. Error message: RATE_LIMITED: ingestion rate limit (local: 15000000 bytes, global: 0 bytes) exceeded while adding 137994 bytes for user xxx

How do I fix this error ?

Steps to reproduce

High concurrency scenarios may occur

Expected behavior

normal

Actual behavior

[otel.javaagent 2024-10-16 17:28:52:560 +0800] [OkHttp http://xxx:4317/...] WARN io.opentelemetry.exporter.internal.grpc.GrpcExporter - Failed to export spans. Server responded with gRPC status code 8. Error message: RATE_LIMITED: ingestion rate limit (local: 15000000 bytes, global: 0 bytes) exceeded while adding 137994 bytes for user xxx

Javaagent or library instrumentation version

https://github.com/open-telemetry/opentelemetry-java-instrumentation/releases/tag/v2.4.0

Environment

JDK: OS:

Additional context

No response

1095071913 commented 1 week ago

Resolved by changing tempo configuration