OTel streams closed unexpectedly

[SantoDE]

Gloo Edge Product

Enterprise

Gloo Edge Version

1.14.6

Kubernetes Version

1.25.X

Describe the bug

A customer is trying to send traces to OpenSearch using the the Data-Prepper component for ingestion. Envoy is exporting traces directly to this component. Data-Prepper seems to expect unary RPC calls for trace exports, in accordance with the OTEL spec. However, it seems that Envoy's implementation is client-side streaming trace exports (The stream is not closed after the export message is sent). This leads to a deframing/decoding issue on the Data-Prepper side (an end-stream flag is never seen, so trace export messages are not handled).

Expected Behavior

No de-framing issues

Steps to reproduce the bug

Setup Otel Tracing exporter as mentioned above, look at data received by tracing backend

Additional Environment Detail

No response

Additional Context

Code in Envoy thats making weird things https://github.com/envoyproxy/envoy/blob/main/source/extensions/tracers/opentelemetry/grpc_trace_exporter.h#L71

Observations

• it seems that a stream is created and closed for each export message. This implies a faulty implementation on Envoy's side. The reference OTEL server, the OTEL collector, is able to process envoy's trace export requests. It looks like it relies on the GRPC message-length flag to identify the end of each message, without relying on the end-stream flag

[github-actions[bot]]

This issue has been marked as stale because of no activity in the last 180 days. It will be closed in the next 180 days unless it is tagged "no stalebot" or other activity occurs.