Closed jonmast closed 2 months ago
Hey,
I do not think we have a great setup yet for AWS + Custom OTEL setup. Just so I understand, what's the reason for going with a custom setup vs. just using @sentry/aws-serverless
? Do you already have a working otel setup? If so, could you share the code of your setup "before sentry"?
If you have a manual setup, you do not need to add @opentelemetry/instrumentation-aws-sdk
or @opentelemetry/instrumentation-aws-lambda
because they are auto-added for you by @sentry/aws-serverless
. If you do not want this, you have to filter this from the default integrations (see https://docs.sentry.io/platforms/javascript/guides/aws-lambda/configuration/integrations/#modifying-default-integrations).
Generally, if you set skipOpenTelemetrySetup: true
the Sentry SDK will not register any tracer etc. for OTEL. So I don't think the error comes from Sentry then 🤔 You can see in the stack trace that this comes from @opentelemetry/sdk-trace-node which we do not even use in the Sentry SDK, this only uses sdk-trace-base.
Do you already have a working otel setup?
Yes. We are using the AWS provided lambda layer from https://aws-otel.github.io/docs/getting-started/lambda/lambda-js, there's no custom setup code on our side. It uses the stock otel layer under the hood with some custom config for AWS. Most of the setup happens in this wrapper.ts file which you can see in the stacktrace. It calls the global.configureTracerProvider
function as an extension mechanism.
Generally, if you set skipOpenTelemetrySetup: true the Sentry SDK will not register any tracer etc. for OTEL. So I don't think the error comes from Sentry then 🤔
You are correct, but what is happening is that the stock initialization in wrapper.ts is running after Sentry's initialization which is resulting in the duplication errors.
OK, I see, I think!
I think the problem is that you are calling provider.register()
again, but it has already been registered before. This is what leads to the duplicate registration issue. So you cannot call register
in there, but must configure the propagator and context manager differently. You can use configureSdkRegistration
for this, I believe, looking at their code. so something like this could work:
import * as Sentry from "@sentry/aws-serverless";
import { NodeTracerProvider } from "@opentelemetry/sdk-trace-node";
import { SentryPropagator, SentrySpanProcessor } from "@sentry/opentelemetry";
import { SentryContextManager, validateOpenTelemetrySetup } from "@sentry/node";
const sentryClient = Sentry.init({
dsn: process.env.SENTRY_DSN,
environment: process.env.SENTRY_ENVIRONMENT,
normalizeDepth: 8,
skipOpenTelemetrySetup: true,
integrations: [Sentry.extraErrorDataIntegration({ depth: 8 })],
});
declare global {
// Global hook exposed by otel layer
function configureTracerProvider(tracerProvider: NodeTracerProvider): void;
function configureSdkRegistration(config: Record<string, any>): Record<string, any>;
function configureInstrumentations(): any[];
function configureTracer(config: any): any;
}
const oldConfigureTracerProvider = global.configureTracerProvider;
const oldConfigureSdkRegistration = global.configureSdkRegistration;
// We let Sentry register instrumentation
global.configureInstrumentation = function() {
return [];
}
global.configureTracer = (config) => {
config.sampler = sentryClient ? new SentrySampler(sentryClient) : undefined,
return config;
}
global.configureSdkRegistration = (config) => {
config.propagator = new SentryPropagator();
config.contextManager = new SentryContextManager();
if (typeof oldConfigureSdkRegistration === 'function') {
return oldConfigureTracerProvider(config);
}
return config;
};
global.configureTracerProvider = (provider) => {
if (typeof oldConfigureTracerProvider === "function") {
oldConfigureTracerProvider(provider);
}
provider.addSpanProcessor(new SentrySpanProcessor());
};
Something like this should work, I believe!
Ah, that makes sense, thanks for the explanation. Your example seems to be working well, thank you!
Closing because this looks solved. Feel free to ping us if there are any more issues!
Is there an existing issue for this?
How do you use Sentry?
Sentry Saas (sentry.io)
Which SDK are you using?
@sentry/aws-serverless
SDK Version
8.13.0
Framework Version
No response
Link to Sentry event
No response
SDK Setup/Reproduction Example
Steps to Reproduce
Expected Result
AWS otel w/ xray and Sentry work together in beautiful harmony :)
I attempted to use the manual setup guide in the docs. It's not entirely clear to me from the docs how Sentry is supposed to fit into an existing opentelemetry setup.
Actual Result
The configs clash and log the following error: