AWS Lambda Problem Node.js & Latest

[SouthBelair]

Environment

SaaS (https://sentry.io/)

Steps to Reproduce

We had an existing aws lambda integration in place and noticed that some of our lambdas were missing from the configure list. In trying to correct this we created a new installation and selected an existing project and aws stack, this updated our lambdas to the "latest". The lambdas immediately began producing errors (see below).

Here's what changed:

Lambda ARN changed from arn:aws:lambda:us-east-1:943013980633:layer:SentryNodeServerlessSDK:184 to arn:aws:lambda:us-east-1:943013980633:layer:SentryNodeServerlessSDK:245

NODE_OPTIONS env value changed from -r @sentry/serverless/dist/awslambda-auto to -r @sentry/aws-serverless/cjs/awslambda-auto

We had to disable sentry on all of our node.js lambdas so that they would continue to operate, in order for us to restore sentry reporting we'll have to manually update every lambda with the old layer and NODE_options values, we have hundreds of lambdas.

node:internal/modules/cjs/loader:597 throw e; ^ Error [ERR_PACKAGE_PATH_NOT_EXPORTED]: Package subpath './cjs/awslambda-auto' is not defined by "exports" in /opt/nodejs/node_modules/@sentry/aws-serverless/package.json at new NodeError (node:internal/errors:405:5) at exportsNotFound (node:internal/modules/esm/resolve:371:10) at packageExportsResolve (node:internal/modules/esm/resolve:718:9) at resolveExports (node:internal/modules/cjs/loader:590:36) at Module._findPath (node:internal/modules/cjs/loader:664:31) at Module._resolveFilename (node:internal/modules/cjs/loader:1126:27) at Module._load (node:internal/modules/cjs/loader:981:27) at internalRequire (node:internal/modules/cjs/loader:174:19) at Module._preloadModules (node:internal/modules/cjs/loader:1569:5) at loadPreloadModules (node:internal/process/pre_execution:628:5) { code: 'ERR_PACKAGE_PATH_NOT_EXPORTED' } Node.js v18.20.2 INIT_REPORT Init Duration: 156.91 ms Phase: init Status: error Error Type: Runtime.ExitError node:internal/modules/cjs/loader:597 throw e; ^ Error [ERR_PACKAGE_PATH_NOT_EXPORTED]: Package subpath './cjs/awslambda-auto' is not defined by "exports" in /opt/nodejs/node_modules/@sentry/aws-serverless/package.json at new NodeError (node:internal/errors:405:5) at exportsNotFound (node:internal/modules/esm/resolve:371:10) at packageExportsResolve (node:internal/modules/esm/resolve:718:9) at resolveExports (node:internal/modules/cjs/loader:590:36) at Module._findPath (node:internal/modules/cjs/loader:664:31) at Module._resolveFilename (node:internal/modules/cjs/loader:1126:27) at Module._load (node:internal/modules/cjs/loader:981:27) at internalRequire (node:internal/modules/cjs/loader:174:19) at Module._preloadModules (node:internal/modules/cjs/loader:1569:5) at loadPreloadModules (node:internal/process/pre_execution:628:5) { code: 'ERR_PACKAGE_PATH_NOT_EXPORTED' } Node.js v18.20.2 INIT_REPORT Init Duration: 634.31 ms Phase: invoke Status: error Error Type: Runtime.ExitError START RequestId: e2961f2a-10cc-4980-affa-a616efaaa7b8 Version: $LATEST RequestId: e2961f2a-10cc-4980-affa-a616efaaa7b8 Error: Runtime exited with error: exit status 1 Runtime.ExitError END RequestId: e2961f2a-10cc-4980-affa-a616efaaa7b8 REPORT RequestId: e2961f2a-10cc-4980-affa-a616efaaa7b8 Duration: 655.34 ms Billed Duration: 656 ms Memory Size: 256 MB Max Memory Used: 15 MB

Expected Result

N/A

Actual Result

N/A

Product Area

Settings - Integrations

Link

No response

DSN

No response

Version

No response

[getsantry[bot]]

Assigning to @getsentry/support for routing ⏲️

[getsantry[bot]]

Routing to @getsentry/product-owners-settings-integrations for triage ⏲️

[fnfilho]

Having the same problem in production today. Had to rollback and remove our Sentry setup.

[andreiborza]

hello, sorry that you've experienced that.

Version 184 of the Lambda layer was still running the v7 version of our SDK.

While it's not a true fix for your problem, you can continue to use a v7 version of the layer. We specifically published one for 7.116.0, please see https://docs.sentry.io/platforms/javascript/guides/aws-lambda/layer/#lambda-layer-for-v7 for the ARN (unfortunately, you need to modify it yourself for your specific region).

Let me know if that helps at all.

[fnfilho]

Hey @andreiborza,

Thank you for following up. I am still not able to use AWS Lambda Layer. Ticket #12012 got me confused as well to what is the right configs to set and I can't seem to find a definitive guide and docs of which Lambda Layer Version refers to what SDK Version.

So this is what I've tried so far, all combinations of the following values. Nothing seems to work.

ARNS: arn:aws:lambda:sa-east-1:943013980633:layer:SentryNodeServerlessSDK:184 arn:aws:lambda:sa-east-1:943013980633:layer:SentryNodeServerlessSDK:240 arn:aws:lambda:sa-east-1:943013980633:layer:SentryNodeServerlessSDK:250 arn:aws:lambda:us-west-1:943013980633:layer:SentryNodeServerlessSDKv7:1

NODE_OPTIONS --require @sentry/aws-serverless/cjs/awslambda-auto --require @sentry/aws-serverless/dist/awslambda-auto --require @sentry/aws-serverless/awslambda-auto

SENTRY_DSN https://08ed8a1c1939e2689dae86933fd192a0@o359692.ingest.us.sentry.io/XXX

SENTRY_TRACES_SAMPLE_RATE 1.0

[paultheurer]

Having this issue as well. Had to turn off sentry lambda layer for all of our lambdas.

Updating to the latest version of the lambda layer via the sentry UI caused this.

[andreiborza]

For arn:...:SentryNodeServerlessSDK:235 and lower, and arn:...:SentryNodeServerlessSDKv7:1 (these use the v7 version of our SDK) the NODE_OPTIONS should include --require @sentry/serverless/dist/awslambda-auto

[fnfilho]

Thank you @andreiborza I can confirm version 235 and NODE_OPTIONS @sentry/serverless/dist/awslambda-auto builds and runs correctly.

I also have set SENTRY_DSN: https://xxx@xxx.ingest.us.sentry.io/xxx and SENTRY_TRACES_SAMPLE_RATE: 1.0

With that said, neither errors nor performance data is being sent to Sentry. Is there a way to debug it better?

[andreiborza]

@fnfilho try setting debug: true and checking your aws logs. If you don't spot anything, please paste it here.

[fnfilho]

Hey @andreiborza How exactly should I set debug: true? Is there an environment variable that has Sentry logs?

I did try by changing the package, but still did not work. This is what I did:

1. Downloaded the 235 version package;
2. Changed the awslambda-auto.js file, adding the debug: true parameter;
3. Uploaded a new zip file as a Custom AWS Layer (file attached);
4. Updated a function with the new custom AWS Layers that runs my own package with the parameter;
5. Ran the function

Logging than said: [Sentry] Cannot initialize SDK with debug option using a non-debug bundle.

sentry7.zip

[andreiborza]

@fnfilho you should be able to set SENTRY_DEBUG=1 in the env variables you provide to your lambda function. Let me know if you can't and I'll have another look.

[fnfilho]

Thank you for following up, @andreiborza. I rolled back to package 235 and had the following env variables set:

NODE_OPTIONS: --require @sentry/serverless/dist/awslambda-auto --enable-source-maps --stack-trace-limit=1000
SENTRY_DEBUG: 1
SENTRY_DSN: https://xxx@xxx.ingest.us.sentry.io/xxx
SENTRY_TRACES_SAMPLE_RATE: 1.0

Than I forced an error to happen by using throw new Error("Sentry test"). No erros captured by Sentry and no logs in AWS about Sentry neither.

[fnfilho]

I tried updating to the new version 252 (arn: arn:aws:lambda:sa-east-1:943013980633:layer:SentryNodeServerlessSDK:252). Can confirm the problem persists.

Error: Error [ERR_PACKAGE_PATH_NOT_EXPORTED]: Package subpath './cjs/awslambda-auto' is not defined by "exports" in /opt/nodejs/node_modules/@sentry/aws-serverless/package.json

Env variables: NODE_OPTIONS: -r @sentry/aws-serverless/cjs/awslambda-auto SENTRY_DSN: https://xxx@xxx.ingest.us.sentry.io/xxx SENTRY_TRACES_SAMPLE_RATE: 1.0

[andreiborza]

@fnfilho that's because all the versions above 235 use the new export via --require @sentry/aws-serverless/awslambda-auto.

[fnfilho]

Hey @andreiborza, I get you have too many tickets to go through and you might not seen all the comments I did and just checked the latest one. So I'll try to summarize it all here.

TRYING WITH V7 ARN: arn:aws:lambda:sa-east-1:943013980633:layer:SentryNodeServerlessSDK:235 NODE_OPTIONS: --require @sentry/serverless/dist/awslambda-auto --enable-source-maps --stack-trace-limit=1000 SENTRY_DEBUG: 1 SENTRY_DSN: https://xxx@xxx.ingest.us.sentry.io/xxx SENTRY_TRACES_SAMPLE_RATE: 1.0

Package Builds Ok. No erros on CloudWatch. No debug messages either. But when I throw an Error in code, this is not captured by Sentry. There are no logs from Sentry at all in CloudWatch.

TRYING WITH V8 ARN: arn:aws:lambda:sa-east-1:943013980633:layer:SentryNodeServerlessSDK:252 NODE_OPTIONS: --require @sentry/aws-serverless/awslambda-auto --enable-source-maps --stack-trace-limit=1000 SENTRY_DEBUG: 1 SENTRY_DSN: https://xxx@xxx.ingest.us.sentry.io/xxx SENTRY_TRACES_SAMPLE_RATE: 1.0

Package won't build. There are a few logs in CloudWatch:

/opt/nodejs/node_modules/@sentry/aws-serverless/build/npm/cjs/index-C86OzNuF.js:2
Node.js v18.20.3
INIT_REPORT Init Duration: 382.11 ms Phase: invoke Status: error Error Type: Runtime.ExitError
START RequestId: 042cf6a4-429a-467a-b727-7e84bf8a9b44 Version: $LATEST
RequestId: 042cf6a4-429a-467a-b727-7e84bf8a9b44 Error: Runtime exited with error: exit status 1 Runtime.ExitError
END RequestId: 042cf6a4-429a-467a-b727-7e84bf8a9b44

It also seems to log the Sentry code. Attached full log export. logs.txt

Side note: when setting up the layer via the Sentry Dashboard in Integrations, it is adding the wrong env variable, it sets --require @sentry/aws-serverless/cjs/awslambda-auto instead of the right one --require @sentry/aws-serverless/awslambda-auto

[andreiborza]

Thank you for the rundown. I'll try to reproduce this tomorrow when I have a bit more time.

As for the Sentry Dashboard, thanks! This should be rolling out soon: https://github.com/getsentry/sentry/pull/73267