Open nlochschmidt opened 1 day ago
8.34.0
only creates a single trace, so it looks like this is a regression in 8.35.0
. The problem is still present in 8.36.0-beta.0
.
Some more things that I tried
HttpIntegration({ spans: false })
: This does remove the @opentelemetry_sentry-patched/instrumentation-http
spans, but now I get a double span for @opentelemetry/instrumentation-http
instead of a single one, and also any captured exceptions no longer have the request
data. So no good.
Keep HttpIntegration
from Sentry, but remove the HttpInstrumentation
from OTEL: The one additional @opentelemetry/instrumentation-http
span goes away and I am left with two @opentelemetry_sentry-patched/instrumentation-http
spans. However captured exceptions still have the request
data
Use HttpIntegration({spans: false})
and remove HttpInstrumentation
from OTEL (combination of the two above): With this the request
data on captured exceptions works, but I no longer get any spans (neither from http
nor from hapi
.
@mydea could this be related to https://github.com/getsentry/sentry-javascript/pull/13763?
The release notes for 8.35.0
mention a SentryHttpInstrumentation
that could be used with OTEL instead of the HttpInstrumentation
:
This change introduces a new SentryHttpInstrumentation to handle non-span related HTTP instrumentation, allowing it to run side-by-side with OTel's HttpInstrumentation.
This sounds promising, however I don't think SentryHttpInstrumentation
is exported from @sentry/node
, and I haven't found documentation on how to use SentryHttpInstrumentation
.
Hy, your guess is right. This change was introduced in 8.35.0
and we still need to update documentation (created an issue). You have to add the following now, so Sentry does not emit spans (only your custom setup).
Sentry.init({
integrations: [Sentry.httpIntegration({ spans: false })]
})
As you wrote you get double spans when setting this to false
: This should not happen and I would have to reproduce this with your example (thanks for providing this).
Hey @nlochschmidt
So after hours of debugging and nearly loosing my mind, we found a solution. In your reproduction, two module.register()
calls were made: one in the hooks.ts
file, you --import
when starting the app, and one in Sentry.init
. I'm fairly confident that this caused the double wrapping of the http
module, which caused duplicated spans. This, paired with Sentry.httpIntegration
being enabled by default and therefore sending yet another root span/trace, brings us indeed to three traces/root spans.
To fix this:
Sentry.httpIntegration({ spans: false })
in your Sentry.init
call. This brings us down to two tracesregister
calls. You have two options:
register
call:
hooks.ts
import "instrument.ts"
line in index.ts
node --import instrument.js index.js
register
call:
hooks.ts
to the top of instrument.ts
import "instrument.ts"
line in index.ts
node --import instrument.js index.js
registerEsmLoaderHooks: false
in your Sentry.init
options.To sum up (we will update the Custom Otel Setup docs accordingly):
Sentry.httpIntegration({spans: false})
for non-span related request handling, like isolating data and attaching request data to errors. This needs to be set explicitly because otherwise, we fall back to Sentry.httpIntegration({spans: true})
which creates another root span.@s1gr1d @Lms24 I was starting to loose my mind as well 😂 Thank you!
I can confirm that this works in the example project (with Sentry register call, with own register call).
It's interesting that the --import ./build/instrument.js
is required to get the hapi instrumentation to work. The http instrumentation by itself works with a simple import "./instrument.js"
at the top of the index.ts
file and no --import
. Very odd behavior.
Will try it in production next 🤞
Http instrumentation works because it's a built-in module. All other instrumentations (node fetch being another exception) like hapi, express, pg, etc. need to be initialized beforehand (via --import loader).
Is there an existing issue for this?
How do you use Sentry?
Sentry Saas (sentry.io)
Which SDK are you using?
@sentry/node
SDK Version
8.35.0
Framework Version
hapi 21.3.10
Link to Sentry event
No response
Reproduction Example/SDK Setup
https://github.com/nlochschmidt/sentry-duplicate-tracing-bug
Steps to Reproduce
curl localhost:3000
I get aspanId
andtraceId
from inside the handlerConsoleSpanExporter
will print the traces and spans collected after a short time.Expected Result
I expect to get a single trace with two spans (one for http and one for hapi).
Actual Result
What actually happens is that there are three traces and only one of them (
traceId: 9dbf48ed9eee5aa66cba36df77eef90b
) has the two spans that are expected, while the other two traces have only a single http root span.I can see some odd instrumentation scopes
@opentelemetry_sentry-patched/instrumentation-http
, but interestingly the@opentelemetry/instrumentation-http
is on one of the traces with a single span.Here is the output cleaned up a bit (full output in the example repo README):
It works as expected when removing Sentry