Keep APM<>RUM link when using OpenTelemetry JS tracer

[bostondv]

In our web client we use RUM browser SDK alongside an OpenTelemetry JavaScript SDK. This OpenTelemetry tracer instruments fetch requests and appends x-datadog-trace-id and x-datadog-parent-id headers to them in order to link the client request spans with the server spans in APM.

However, when RUM is also enabled in our web client, RUM merges the values of the x-datadog-* headers with its own resulting both values separated by comma. This breaks span linkage on both our APM traces and RUM.

"x-datadog-origin": "opentelemetry-sdk-web, rum",
"x-datadog-parent-id": "12336011929620396511, 8863242246433125602",
"x-datadog-sampled": "1, 1",
"x-datadog-sampling-priority": "1, 1",
"x-datadog-trace-id": "12729276616416176546, 292731172712202065"

Could you consider adding a check so that if any of these headers are already present in the request to no overwrite them?

Thanks 🙇

@datadog/browser-rum-slim v4.6.1

[BenoitZugmeyer]

Thanks for your feedback. This seems reasonable, we'll see what we can do!

[BenoitZugmeyer]

@bostondv: any reason to enable tracing in the RUM SDK if you are already using another SDK for it?

[bostondv]

@BenoitZugmeyer We're experimenting with the other SDK. It might become a replacement for RUM or we might opt to keep both depending on if one SDK can fulfill all our observability needs.

Setting allowedTracingOrigins to an empty array or completely disabling RUM when our other SDK is enabled is for sure a potential workaround. But if we opt to run both alongside each other it would be ideal to keep SDK integrations with APM working.

Given that RUM generates a random trace and span ID for these headers I'm thinking that if RUM retains existing headers, when present, that both SDK integrations with APM may be able to work together?

wdyt?

[BenoitZugmeyer]

The OpenTelemetry SDK will generate a trace itself, right?

• Does it send it to Datadog as well? If so, both the RUM SDK and the other SDK are generating the same trace, do you expect it to show up twice in the Trace explorer? I'm not even sure if it's possible.

• Or does it send the trace to another service, in which case why using x-datadog-* headers for it?

I'm asking because your use case, while valid, seems to be a bit niche, and implementing what you ask for is a bit tricky especially because there is no xhr.getRequestHeader to check if the header is already present or not.

[bostondv]

We're using the OpenTelemetry SDK to create traces on the web client. Within the trace, among other things, we instrument fetch to create spans for each fetch request. For these instrumented fetch requests we add x-datadog-parent-id and x-datadog-trace-id headers so that backend service spans are included in the trace generated by OpenTelemetry SDK when viewing it in APM.

What I'm hoping to do is maintain the backend trace link that shows up in RUM when looking a view waterfalls and resources as well as make the link to backend spans in our OpenTelemetry traces when viewing them in APM.

I totally understand its a bit of niche request. With fetch it should be possible to check the request headers but agree with you on xhr, not sure what you'd do there.

[bcaudan]

Hi @bostondv,

What I'm hoping to do is maintain the backend trace link that shows up in RUM when looking a view waterfalls and resources as well as make the link to backend spans in our OpenTelemetry traces when viewing them in APM.

For your use case, a target integration could be to have:

• open telemetry SDK tracing requests and sending spans to datadog servers
• RUM browser SDK not tracing requests but attaching somehow the trace id generated by open telemetry to RUM events

We definitely plan to have a better integration with open telemetry so we want to address this use case but it is not in our current priorities right now.

We will keep you posted here when we will make some progress on this topic.

[leandrorebelo]

I'm waiting for this...

[sodabrew]

@bostondv

We're using the OpenTelemetry SDK to create traces on the web client. Within the trace, among other things, we instrument fetch to create spans for each fetch request. For these instrumented fetch requests we add x-datadog-parent-id and x-datadog-trace-id headers so that backend service spans are included in the trace generated by OpenTelemetry SDK when viewing it in APM.

Do you mean that you're using OpenTelemetry-JS on the client side and Datadog APM libraries on the server side? In that case, why not use Datadog on both ends - that is, what's the upside of OpenTelemetry-JS on the client side?

Conversely, I've found that OpenTelemetry on the server side is a valuable choice in favor of open standards to avoid lock-in, while vendors still have a functional lead on the browser side. In this case, linking spans from RUM in browser -> OpenTelemetry-{Go, Node, Java, etc.} on the backend is required. Header munging from x-vendor headers to W3C standards is the current best workaround, but native compatibility as a feature rquest e.g. #1639 would be nice.

[naseemkullah]

I believe this can be closed via https://github.com/DataDog/browser-sdk/pull/1832 which allows browser-sdk to inject w3c context

[bcaudan]

Hi, we will ping this thread with some details when it will be released and indeed close this ticket.

[yannickadam]

Hi @bostondv

What I'm hoping to do is maintain the backend trace link that shows up in RUM when looking a view waterfalls and resources as well as make the link to backend spans in our OpenTelemetry traces when viewing them in APM.

This feature is now available through https://github.com/DataDog/browser-sdk/pull/1832

However, we're keeping this issue opened as the initial topic - compatibility with OpenTelemetry libraries - still needs to be resolved.

[royteeuwen]

Could I perhaps get some extra context?

I have an application with on server-side the otel java agent, and an otel collector to send to datadog.

i’d like to now include the opentelemetry js web sdk to also have traces clientside. Can I currently do this with datadog, and if yes, what endpoint should be used? The collector or can we send directly to datadog?

am i forced to also include datadog js libs clientside, because we definitely want to avoid this

[bcaudan]

Hi @royteeuwen,

the configuration of the opentelemetry setup is outside the expertise of this repository, I'd advise you to reach out to datadog support to get more help on this topic.