sentry_cpu_profiler-v83-linux-arm64-glibc.node: cannot open shared object file: No such file or directory

[ex-nerd]

Is there an existing issue for this?

• [X] I have checked for existing issues https://github.com/getsentry/profiling-node/issues
• [X] I have reviewed the documentation https://docs.sentry.io/
• [X] I am using the latest SDK release https://github.com/getsentry/profiling-node/releases

How do you use Sentry?

Sentry Saas (sentry.io)

SDK Version

7.36.0

Link to Sentry event

n/a

What environment is your node script running in?

AWS Lambda, Node.js 14.x, arm64

How is your code deployed and bundled?

Lambda package deploy (zip file bundling our code, node_modules, etc)

Steps to Reproduce

Custom code wrapping lambda, looks something like this:

export function wrapLambdaHandler<TEvent, TResult>(
    dsn: string,
    handler: Handler<TEvent, TResult>,
    options: IWrapperOptions = {},
): Handler<TEvent, TResult> {
    const integrations: Integration[] = [];
    const initOptions: Sentry.NodeOptions = {
        dsn: dsn,
        environment: 'stage',
        tracesSampleRate: options.tracesSampleRate ?? 0.1,
        beforeSend: beforeSendToSentry,
        ignoreErrors: ['ProvisionedThroughputExceededException', 'TimeoutError', /Possible function timeout/],
        integrations: [],
    };
    const profilesSampleRate = options.profilesSampleRate ?? 0.1;
    if (profilesSampleRate > 0) {
        initOptions.profilesSampleRate = profilesSampleRate;
        initOptions.integrations.push(new ProfilingIntegration());
    }
    AWSLambda.init(initOptions);
    return AWSLambda.wrapHandler(handler);
}

Stack trace (below) doesn't really show if the failure happens during the library load, or during the init phase.

Expected Result

A functional lambda.

Actual Result

Lambda fails with /var/task/node_modules/@sentry/profiling-node/binaries/sentry_cpu_profiler-v83-linux-arm64-glibc.node: cannot open shared object file: No such file or directory

The full error looks like:

{
  "errorType": "Error",
  "errorMessage": "/var/task/node_modules/@sentry/profiling-node/binaries/sentry_cpu_profiler-v83-linux-arm64-glibc.node: cannot open shared object file: No such file or directory",
  "code": "ERR_DLOPEN_FAILED",
  "stack": [
    "Error: /var/task/node_modules/@sentry/profiling-node/binaries/sentry_cpu_profiler-v83-linux-arm64-glibc.node: cannot open shared object file: No such file or directory",
    "    at Object.Module._extensions..node (internal/modules/cjs/loader.js:1144:18)",
    "    at Module.load (internal/modules/cjs/loader.js:950:32)",
    "    at Function.Module._load (internal/modules/cjs/loader.js:790:12)",
    "    at Module.require (internal/modules/cjs/loader.js:974:19)",
    "    at require (internal/modules/cjs/helpers.js:101:18)",
    "    at importCppBindingsModule (/var/task/node_modules/@sentry/profiling-node/lib/cpu_profiler.js:20:12)",
    "    at Object.<anonymous> (/var/task/node_modules/@sentry/profiling-node/lib/cpu_profiler.js:26:25)",
    "    at Module._compile (internal/modules/cjs/loader.js:1085:14)",
    "    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1114:10)",
    "    at Module.load (internal/modules/cjs/loader.js:950:32)"
  ]
}

I've downloaded the .zip file that was deployed to Lambda and can confirm that the sentry_cpu_profiler-v83-linux-arm64-glibc.node file does exist, so this appears to be a problem internal to the lambda itself.

Unfortunately, since this is a problem with Sentry initialization, I don't have a Sentry issue with more detailed info. 😉

[JonasBa]

@ex-nerd, can you verify that bundling and execution arch are the same? This could happen if you run bundling on a glibc based image, but execute on a musl based image (or vice versa)

[JonasBa]

This could be related to this documentation.

Deployment packages with C or C++ libraries If your deployment package contains native libraries, you can build the deployment package with AWS Serverless Application Model (AWS SAM). You can use the AWS SAM CLI sam build command with the --use-container to create your deployment package. This option builds a deployment package inside a Docker image that is compatible with the Lambda execution environment.

[ex-nerd]

I'm just poking this to let you know that "roadmap work" has prevented me from doing any further debugging on this. The issue isn't yet resolved, but I also don't have any new information.

[JonasBa]

@ex-nerd Thanks for the updated. To be fair, I have not had the chance to actually attempt to build a package for aws lambdas myself. Let us know once you get back around to debugging this - I will update here in case we get to it first.

[ex-nerd]

Just checking in here again. Bundling and execution arch are not the same, and we will eventually get that migrated … however, the lambda is being invoked from a .zip file stored in S3. That file is confirmed to contain the appropriate binary for the arch (and all of the other ones), so any help in debugging this would be appreciated.

[ex-nerd]

I'll also add that we're deploying via cloudformation, e.g. (semi-redacted)

{
    "Type": "AWS::Lambda::Function",
    "Properties": {
        "Architectures": ["arm64"],
        "FunctionName": "LambdaName",
        "Description": "LambdaDescription",
        "Runtime": "nodejs14.x",
        "Handler": "lambda.handler",
        "Role": "LambdaRoleArn",
        "Code": {
            "S3Bucket": "S3_BUCKET_NAME_GOES_HERE",
            "S3Key": "LAMBDA_CODE_PATH.zip"
        },
        "Environment": {
            "Variables": {
                "NODE_OPTIONS": "--enable-source-maps",
                "SENTRY_CLOUD_DSN": "3.14159265359",
                "SENTRY_TRACES_SAMPLE_RATE": "0.1",
            }
        }
    }
}

docs for AWS::Lambda::Function available here

There doesn't seem to be a way for us to specify the build architecture of the lambda other than what's shown here. I'm still digging, just thought I'd send this update in case it sparks an idea.

[JonasBa]

@ex-nerd I will look into this today as there have been a few reports, thanks for the added info

[JonasBa]

@ex-nerd I have not setup the lambda deploys, but I suspect the cause here could be a missing shared library like libstdc. We've had a couple of reports of the same nature in different runtimes so I'm trying to see what the cause could be here.

Out of curiosity, does the same error happen if you attempt to use node 16 or 18?

[ex-nerd]

I don't currently have a way to test lambda with node 16 or 18, sorry.

[yasircodingcrafts]

Hi, I am having this same type of error but on an ec2 instance. I am using pm2. There are 4 instances but only one is giving the following error: Error: Cannot find module '/{APPLICATION_NAME}/node_modules/@sentry/profiling-node/lib/sentry_cpu_profiler-linux-x64-glibc-72.node'

[JonasBa]

@yasirtrimulabs you are using nodejs v12 which has been end of life for almost 3 years. We do not provide precompiled binaries for this node version so we would recommend you to update to node v16 or 18 which is the current LTS.

Since only one of the instances gives this error, is it possible that this instance is the only one running node v12 while other run newer versions of node?

With all that said, you should be able to use the package with node v12, but you need to ensure that the binary can be properly compiled from source after installation. If upgrading your node version is not possible, can you please tell me how the package is installed? Additionally, can you please cd into '/{APPLICATION_NAME}/node_modules/@sentry/profiling-node/ and run npm run build:configure and npm run build:bindings and tell us if you see any errors?

[yasircodingcrafts]

@JonasBa thanks for the quick reply. I am using node v16 already. I have set v16 as default using nvm. Can it be that the pm2 is overwriting it?

Here's the output for npm run build:configure

> @sentry/profiling-node@1.2.1 build:configure
> node-gyp configure

gyp info it worked if it ends with ok
gyp info using node-gyp@9.3.0
gyp info using node@16.14.0 | linux | x64
gyp info find Python using Python version 3.10.12 found at "/usr/bin/python3"
gyp http GET https://nodejs.org/download/release/v16.14.0/node-v16.14.0-headers.tar.gz
gyp http 200 https://nodejs.org/download/release/v16.14.0/node-v16.14.0-headers.tar.gz
gyp http GET https://nodejs.org/download/release/v16.14.0/SHASUMS256.txt
gyp http 200 https://nodejs.org/download/release/v16.14.0/SHASUMS256.txt
gyp info spawn /usr/bin/python3
gyp info spawn args [
gyp info spawn args   '/{APPLICATION_NAME}/node_modules/node-gyp/gyp/gyp_main.py',
gyp info spawn args   'binding.gyp',
gyp info spawn args   '-f',
gyp info spawn args   'make',
gyp info spawn args   '-I',
gyp info spawn args   '/{APPLICATION_NAME}/node_modules/@sentry/profiling-node/build/config.gypi',
gyp info spawn args   '-I',
gyp info spawn args   '/{APPLICATION_NAME}/node_modules/node-gyp/addon.gypi',
gyp info spawn args   '-I',
gyp info spawn args   '/root/.cache/node-gyp/16.14.0/include/node/common.gypi',
gyp info spawn args   '-Dlibrary=shared_library',
gyp info spawn args   '-Dvisibility=default',
gyp info spawn args   '-Dnode_root_dir=/root/.cache/node-gyp/16.14.0',
gyp info spawn args   '-Dnode_gyp_dir=/{APPLICATION_NAME}/node_modules/node-gyp',
gyp info spawn args   '-Dnode_lib_file=/root/.cache/node-gyp/16.14.0/<(target_arch)/node.lib',
gyp info spawn args   '-Dmodule_root_dir=/{APPLICATION_NAME}/node_modules/@sentry/profiling-node',
gyp info spawn args   '-Dnode_engine=v8',
gyp info spawn args   '--depth=.',
gyp info spawn args   '--no-parallel',
gyp info spawn args   '--generator-output',
gyp info spawn args   'build',
gyp info spawn args   '-Goutput_dir=.'
gyp info spawn args ]
gyp info ok 

And here's the output for npm run build:bindings

> @sentry/profiling-node@1.2.1 build:bindings
> node-gyp build && node scripts/copy-target.mjs

gyp info it worked if it ends with ok
gyp info using node-gyp@9.3.0
gyp info using node@16.14.0 | linux | x64
gyp info spawn make
gyp info spawn args [ 'BUILDTYPE=Release', '-C', 'build' ]
make: Entering directory '/{APPLICATION_NAME}/node_modules/@sentry/profiling-node/build'
  CXX(target) Release/obj.target/sentry_cpu_profiler/bindings/cpu_profiler.o
  SOLINK_MODULE(target) Release/obj.target/sentry_cpu_profiler.node
  COPY Release/sentry_cpu_profiler.node
make: Leaving directory '/{APPLICATION_NAME}/node_modules/@sentry/profiling-node/build'
gyp info ok 
Target file already exists, overwriting it
Renaming /{APPLICATION_NAME}/node_modules/@sentry/profiling-node/build/Release/sentry_cpu_profiler.node to /{APPLICATION_NAME}/node_modules/@sentry/profiling-node/lib/sentry_cpu_profiler-linux-x64-glibc-93.node

[JonasBa]

@yasirtrimulabs yes, that is possible. If you would have installed with node v12, then the binaries would have been built for that version (or you would have seen an install error)

The reason I know you are using node v12 at runtime is because the binary is looking for sentry_cpu_profiler-linux-x64-glibc-72.node meaning you are have x64 linux arch and node version 72 of nodejs ABI. You can cross reference version 72 at https://nodejs.org/en/download/releases that this points to node v12

So it is likely that something is running your process with node v12 here

[yasircodingcrafts]

@JonasBa thanks for helping. For some reason, the pm2 on this instance was using node v12. It is working now. Thanks again, you are a life saver