hapi traces no longer respects x-datadog-trace-id and x-datadog-parent-id when updating from 2.3.0 to 2.33.0

[jcjones1515]

Expected behaviour

Requests from the browser containing headers set by @datadog/browser-rum should link together requests with server side tracing like this:

Actual behaviour

After updating to 2.33.0 I now get nothing linked:

I observed that the headers are still being sent in dev tools, and no other changes were made.

Steps to reproduce

I see a test to cover this specific scenario, but literally I deployed the same code where the only difference is v2.3.0 -> v2.33.0.

Environment

• Operation system:
• Node.js version: 16
• Tracer version: 2.33.0
• Agent version: latest
• Relevant library versions:

[jcjones1515]

if someone could point me in the right direction I'd be happy to contribute a fix

[prescottprue]

I am seeing a similar issue this with the graphql plugin in versions of dd-trace past v2.1.1 including v3 and v4. A few things I've noticed while looking around:

• I believe dd-trace currently marks spans as express in v3 of Apollo due to express being used internally to apollo-server (could be related to the described issue since it is another request framework like hapi?) - v2.2.0 release mentions > add express parsed body support (https://github.com/DataDog/dd-trace-js/pull/1782)
• We are seeing a different trace id starting at the service and never actually see the trace id from the browser request in service traces. There are now two separate traces, one in our UI, one in our service (separate ids). From the Apollo service to other HTTP services running other HTTP frameworks, there is no issue preserving the trace_id
• v4 of @apollo/server (new version of apollo-server) appears to no longer use express internally, maybe that will fix the issue? Not sure - I'll post again if I find that it does, but this is a big change on our end
• v3.14.0 release mentions adding support for W3C trace context headers and this seems to be where things break - are clients supposed to switch RUM config to pass tracecontext for W3C trace-context headers instead of DD default? I haven't tried this, but if it is the case my assumption would be that would be a breaking change in dd-trace to switch to requiring new headers?

Here are the versions I've tried and what preserves trace_id from browser requests through a service running Apollo (graphql plugin) and what does not:

v2.1.1 - works
v2.2.0 - works
v3.0.0 - works
v3.10.0 - works
v3.11.0 - works
v3.12.0 - works
v3.13.0 - works
v3.14.0 - does not work
v3.16.0 - does not work
v3.19.0 - does not work
v4.0.0 - does not work

Operation system: Both mac and linux
Node.js version: ^18
Tracer version: 4
Agent version: latest
Relevant library versions: ^3.14.0, ^4.0.0
ESM: No
Libraries: apollo-server ^3 (internally uses express)
Plugins: graphql 

Happy to provide any more info if needed

[prescottprue]

Workaround

I was able to get things working again with ^3.19.0 and v4 by changing our front end config for RUM to include the datadog and tracecontext in the allowedTracingUrls.*.propagatorTypes like so:

const apiUrl = 'https://ourapi.example.com'

datadogRum.init({
  // other dd rum config
   allowedTracingUrls: [
-      apiUrl
+      {
+        match: apiUrl
+        propagatorTypes: ['datadog', 'tracecontext'],
+      },
    ],
])

I would still say this is a workaround since it isn't using defaults anymore (which don't work) and was "breaking" while being in a minor change

Theory

Since passing the W3C traceparent header gets things to work again I'm wondering if it is that the new logic in v3.14.0 for supporting this header is to blame for the datadog headers no longer being preserved 🤷‍♂️

Note On Previous Post

When posting that things were working above I forgot to include that having different versions of dd-trace in another service which is being called is still dropping the trace id unless all are on the older version

[jcjones1515]

@prescottprue your change seems to have worked for me, I'm seeing it work on 4.1.0