Deploy lambda with muhammaraJS from Mac M1 using serverless

[m-rahme]

Hey @julianhille,

I'm currently trying to figure out a way to deploy my lambda (that contains MuhammaraJS) from my machine (mac m1) to AWS using serverless.

The way my lambda gets deployed on all my environments is the same, with serverless and serverless-esbuild where before every deployment npm install runs and everythings gets packaged into a .zip and then deployed.

I was able to successfully deploy the lambda to my staging, demo, and prod environments using CircleCi and serveless. The reason this worked is because the OS in the CI is the same as the lambda's AWS Linux 2 so the correct binary get's installed. (Note that the runtime is Node.js 16.x)

So for my dev/test environment, since I don't use the CI but rather my own machine (mac m1) the wrong binary is being deployed to the lambda which results in an errorMessage": "/var/task/node_modules/muhammara/binding/muhammara.node: invalid ELF header".

I cannot find a way to run any of these two commands using serverless while my lambda is packaged:

EXTRA_NODE_PRE_GYP_FLAGS="--target_platform=linux" npm install

npx node-pre-gyp install --update-binary --target-platform=linux

Do you have an idea on how I can resolve this and make the packaging and deployment with serverless from my local machine work?

Here's a snippet from serverless.yml:

[julianhille]

before every deployment npm install runs and everythings gets packaged into a .zip and then deployed.

So to my understanding packaging runs on your M1 or on a serverless / other aws instance?

[m-rahme]

For our local DEV environment the packaging runs on the M1 machine and then it's deployed to the lambda.

For our other environments, the packaging runs on CircleCi instance, which happens to be the same as the AWS Lambda, and then it get's deployed. This is why Muhammara works in these environments but not on DEV.

[julianhille]

Are you able to do an md5 or Sha or other hash on the file before and after you run the npx or npm install with extra flags?

And also please tell me which version are you using.

[m-rahme]

I'm using muhammara 3.8.0.

I believe I cannot run any hash since I'm using serverless esbuild to install my npm dependencies before deploying to the lambda. (Check screenshot above)

Thank you.

[julianhille]

your commands from above do not work anymore try this: npm rebuild muhammara --target-platform=darwin --update-binary --target_libc=unknown

and use the needed targets / libcs etc

[julianhille]

Did that help?

[m-rahme]

The problem is that I cannot run this command npm rebuild using serverless esbuild (which is what i'm using to install muhammara in my lambda)

[m-rahme]

I can add the extra arguments but only for npm install as follows:

[julianhille]

ok ok, restart. i thought you run the build process on your local M1 machine? Are you packaging the node_modules locally or remotely on the aws machine?

[m-rahme]

We have 4 different lambdas for 4 different environments.

3 out of those 4 lambdas we package and deploy (using serverless) on CircleCi's VMs running Linux environments. We don't have problems with Muhammara in those lambdas since the environment in CircleCi and the lambda match.

The last lambda, we package and deploy (using serverless) on our local machines which happen to be Mac M1's. This is the lambda we are having problems with since the environment of our machines and the lambda do not match.

Now to include muhammara as well as other libraries in our lambda package before deploying we use "serverless esbuild" inside serverless.yml file (screenshot above) which builds those libraries and adds them to the package.

[julianhille]

ok before you are letting esbuild run, and AFTER a local npm install try the command from above. npm rebuild ... etc

[julianhille]

if that does not work give me shot through mail

[m-rahme]

yeah that does not work. esbuild still builds the wrong binary.

What's your email?

[m-rahme]

what should --target_libc be equal to to be able to run on lambda with AWS Linux 2?

[julianhille]

target lib should be removed I guess your aws is simple libc not musl

[m-rahme]

ok before you are letting esbuild run, and AFTER a local npm install try the command from above. npm rebuild ... etc

I was able to rebuild and deploy. I checked the new muhammara.node and it is indeed different that the previous one that was being compiled so I know that the rebuild command did actually work.

However, even though --target_platform=linux was used. I still am receiving the invalid ELF header error in my lambda.

[julianhille]

Are you able to post the elf header error? Maybe it gives hints. Can you attach the muhammara.node file? Can you link to the specific aws serverless env? I've learned that there are arm machines out there sometime ago.

[m-rahme]

Here's the error message:

2023-08-31T19:20:11.895Z    undefined   ERROR   Uncaught Exception  
{
    "errorType": "Error",
    "errorMessage": "/var/task/node_modules/muhammara/binding/muhammara.node: invalid ELF header",
    "code": "ERR_DLOPEN_FAILED",
    "stack": [
        "Error: /var/task/node_modules/muhammara/binding/muhammara.node: invalid ELF header",
        "    at Object.Module._extensions..node (node:internal/modules/cjs/loader:1282:18)",
        "    at Module.load (node:internal/modules/cjs/loader:1076:32)",
        "    at Function.Module._load (node:internal/modules/cjs/loader:911:12)",
        "    at Module.require (node:internal/modules/cjs/loader:1100:19)",
        "    at require (node:internal/modules/cjs/helpers:108:18)",
        "    at Object.<anonymous> (/var/task/node_modules/muhammara/lib/muhammara.js:6:35)",
        "    at Module._compile (node:internal/modules/cjs/loader:1198:14)",
        "    at Object.Module._extensions..js (node:internal/modules/cjs/loader:1252:10)",
        "    at Module.load (node:internal/modules/cjs/loader:1076:32)",
        "    at Function.Module._load (node:internal/modules/cjs/loader:911:12)"
    ]
}

Here's the environment we're using on CircleCi that actually builds the correct Muhammara version that works on 3 out of 4 of our lambdas.

[m-rahme]

Here's the lambda environment :

[julianhille]

Ok I'm building the pre builts on an Ubuntu x86 with glibc. That should not at all end in an invalid elf header. Please attach several muahammara.node files.

After you do an npm install. After npm rebuild command and after es build is done. Please also attach exact npm rebuild command

[julianhille]

Please contact me through mail maybe we get to the bottom.of it a bit faster there.

[m-rahme]

Here's the muhammra.node that is being built locally after running the rebuild command from above:

npm rebuild muhammara --update-binary --target-platform=linux

https://we.tl/t-3vq2xDglLM

And here's the muhammara.node that is being built on CircleCi (that is currently working perfectly fine):

https://we.tl/t-2GXlSnbjuK

[m-rahme]

Yes I will email you. Thanks.

[julianhille]

fix:

npm rebuild muhammara --target-platform=linux --update-binary --target_libc=glibc --target_arch=x64 --target=16.0.0

have a great day