search

open-telemetry / opentelemetry-specification

Specifications for OpenTelemetry
https://opentelemetry.io
Apache License 2.0
3.74k stars 887 forks source link

Enable load balancing for gRPC based exporters in auto-instrumentation #2212

Open pavolloffay opened 2 years ago

pavolloffay commented 2 years ago

What are you trying to achieve?

I want to load balance OTLP gRPC export requests from auto-instrumentation (e.g. javaagent) to multiple collectors. At the moment there is no way to configure round_robin load balancer in auto-instrumentations, It can be only configured explicitly in code when creating e.g. OTLP exporter.

By default, gRPC client-side load balancing uses pick_first load balancer which chooses the first address and it uses it until it is disconnected. https://grpc.github.io/grpc/core/md_doc_load-balancing.html

What did you expect to see?

Each collector receiving approximately the same amount of data.

Additional context.

Add any other context about the problem here. If you followed an existing documentation, please share the link to it.

pavolloffay commented 2 years ago

The title says auto-instrumentation, but having the support directly in the SDK would be preferred.

anuraaga commented 2 years ago

For reference, the most recent version of the javaagent dropped the dependency on grpc-java, instead hand crafting HTTP/2 requests with the gRPC wire protocol. So business logic related to grpc load balancing has been removed complete. If there is an expectation for load balancing it can be restored. But OTLP's focus is ok the HTTP exporter from what I've seen, and some languages will only support HTTP, for reasons similar to why we dropped the grpc-java dependency.

Defining load balancing more generally for both transports rather than only one is probably more aligned with OTLP. Pragmatically, I guess a sidecar collector would be the scalable way to do load balancing for both transports, for various service discovery mechanisms like DNS, consul, etc.

pavolloffay commented 2 years ago

. But OTLP's focus is ok the HTTP exporter from what I've seen, and some languages will only support HTTP, for reasons similar to why we dropped the grpc-java dependency.

Could you please comment more why dependency on grpc-java was removed? Is HTTP exporter is preferred over gRPC?

anuraaga commented 2 years ago

We still support the gRPC exporter and it's the default - but we don't use the grpc-java library. One of the reasons was an awkward user experience due to the library's pluggable transports, but also to reduce dependency overhead. More details in my "blog post" - https://github.com/open-telemetry/opentelemetry-specification/discussions/1996