sharp is not working ( install ) in aws lambda ( serverless-webpack )

[ghost]

[I used]
1. serverless framework ( aws lambda, aws-nodejs-typescript template )
2. node version 12.14.1
3. sharp version 0.25.2 ( I did test 0.25.0 ~ 0.25.3 )

I want to resize my lambda zip size.

So I did install serverless-webpack plugins and bundle my lambda.

Before bundled by webpack, only transpile typescript to javascript and deploy using serverless and works well.

But, lambda bundled by serverless-webpack is not working.

[Error message] darwin-x64' binaries cannot be used on the 'linux-x64' platform. Please remove the 'node_modules/sharp/vendor'

I don't know how to do.

I did check node_modules/sharp/vendor and saw linux-64 target.

but lambda is darwin-x64 logged.

I don't want to use lambda layer.

How to do?

I did

1. Delete node_modules and package-lock.json and install dependencies ( also installed sharp )
2. Delete node_modules/sharp and install sharp ( lambda environment - linux, x64, node version )
3. Set serverless-webpack configuration in serverless : packagerOptions ( scrips ) - rebuild sharp lambda enviroment But, lambda is not working

upload and delete lambda work well and resized zip size!!

but get ( resize image ) lambda is not.

Please help me.

Thank you

[lovell]

Did you see the following?

https://sharp.pixelplumbing.com/install#aws-lambda

https://github.com/serverless-heaven/serverless-webpack#node-modules--externals

[ghost]

Yes, I did it.

rm -rf node_modules/sharp
npm install --arch=x64 --platform=linux --target=12.14.1 sharp

I did run above command in npm scripts ( before sls deploy )

also, I did

custom:
  webpack:
     packagerOptions:
         scripts:
            - npm rebuild --arch=x64 --platform=linux --target=12.14.1 sharp

And I checked my node_modules/sharp and version information. ( = linux )

But. lambda logged "darwin-64" not "linux-x64".

Additionally,

rm -rf node_modules/sharp
npm install --arch=x64 --platform=linux --target=12.14.1 sharp

This command works well in lambda without serverless-webpack.

But, lambda with serverless-webpack is not working.

[lovell]

Hi, the commands documented at https://sharp.pixelplumbing.com/install#aws-lambda for you to use e.g. in the scripts directive are:

rm -rf node_modules/sharp
npm install --arch=x64 --platform=linux sharp

Please note this is not the same as npm rebuild and there is no --target flag.

[ghost]

Yes, I also did it.

ex1.
rm -rf node_modules/sharp && npm i --arch=x64 --platform=linux sharp

ex2.
rm -rf node_modules/sharp && npm i --arch=x64 --platform=linux -- target=12.14.1 sharp

ex3.
npm rebuild --arch=x64 --platform=linux --target=12.14.1 sharp

But, above examples is not working.

I think that serverless-webpack plugin-in with sharp is not working properly.

Another issue repo said that using lambda layer.

But, I don't want to use lambda layer.. sorry..

[lovell]

Please can you provide a complete, standalone example with all dependencies, code and config, perhaps as a separate repo, that would allow someone else to reproduce this.

[coreysnyder04]

I'm getting this same issue. I pretty much followed this example and then added sharp via this AWS Example.

I wonder if the build layer portion of this is what is screwing things up because it runs:

#!/bin/bash
set -eo pipefail
mkdir -p lib/nodejs
rm -rf node_modules lib/nodejs/node_modules
npm install --production
mv node_modules lib/nodejs/

Which may be installing the wrong version of sharp. When you run npm install --arch=x64 --platform=linux sharp does it save something to the package.json file or the lock file to say use this exact config when you run npm install again? Or does it go and grab whatever is best for that particular build environment?

It certainly doesn't seem like it saves enough info for the latter npm install to know which special version to use.

"dependencies": {
    "aws-xray-sdk-core": "1.1.2",
    "sharp": "^0.25.3"
  },

I tried to update the build layer to

#!/bin/bash
set -eo pipefail
mkdir -p lib/nodejs
rm -rf node_modules lib/nodejs/node_modules
npm install --production
rm -rf node_modules/sharp
npm install --arch=x64 --platform=linux sharp --save
mv node_modules lib/nodejs/

And then run the deploy script.

Deploy Script

#!/bin/bash
set -eo pipefail
ARTIFACT_BUCKET=$(cat bucket-name.txt)
aws cloudformation package --template-file template.yml --s3-bucket $ARTIFACT_BUCKET --output-template-file out.yml
aws cloudformation deploy --template-file out.yml --stack-name rr-image-service --capabilities CAPABILITY_NAMED_IAM

Unfortunately, this results in a new error I've never seen before:

$ ./3-deploy.sh

Unable to upload artifact lib/. referenced by ContentUri parameter of libs resource.
ZIP does not support timestamps before 1980

Here's the full output:

$ ./2-build-layer.sh

> sharp@0.25.3 install /my-service/node_modules/sharp
> (node install/libvips && node install/dll-copy && prebuild-install --runtime=napi) || (node-gyp rebuild && node install/dll-copy)

info sharp Using cached ~/.npm/_libvips/libvips-8.9.1-darwin-x64.tar.gz
added 94 packages from 179 contributors and audited 611 packages in 10.79s

5 packages are looking for funding
  run `npm fund` for details

found 0 vulnerabilities

  > sharp@0.25.3 install /my-service/node_modules/sharp
> (node install/libvips && node install/dll-copy && prebuild-install --runtime=napi) || (node-gyp rebuild && node install/dll-copy)

info sharp Using cached ~/.npm/_libvips/libvips-8.9.1-linux-x64.tar.gz
+ sharp@0.25.3
added 519 packages from 462 contributors and audited 611 packages in 17.998s

13 packages are looking for funding
  run `npm fund` for details

found 0 vulnerabilities

$ ./3-deploy.sh

Unable to upload artifact lib/. referenced by ContentUri parameter of libs resource.
ZIP does not support timestamps before 1980

[ghost]

Thank you.

[package.json]

 "scripts": {
    "dev": "sls offline start",
     "build:prod": "rm -rf node_modules/sharp && npm install --arch=x64 --platform=linux sharp",
    "deploy:dev": "npx serverless deploy --aws-profile "profile" --stage dev --env development",
    "deploy:prod": "npx serverless deploy --aws-profile "profile" --stage prod --env production"
  },
 "dependencies": {
    "busboy": "^0.3.1",
    "http-errors": "^1.7.3",
    "sharp": "^0.25.2",
    "tslib": "^1.11.1",
    "uuid": "^7.0.3"
  },
  "devDependencies": {
    "@types/http-errors": "^1.6.3",
    "@types/sharp": "^0.25.0",
    "@types/aws-lambda": "^8.10.17",
    "@types/busboy": "^0.2.3",
    "@types/node": "^10.12.18",
    "@types/uuid": "^7.0.3",
    "aws-lambda": "^1.0.6",
    "aws-sdk": "^2.685.0",
    "serverless-dotenv-plugin": "^2.4.2",
    "serverless-offline": "^6.1.4",
    "serverless-webpack": "^5.2.0",
    "ts-loader": "^5.3.3",
    "typescript": "^3.2.4",
    "webpack": "^4.29.0",
    "webpack-node-externals": "^1.7.2"
  },

[serverless.yml]

custom:
  webpack:
    webpackConfig: ./webpack.config.js
    includeModules: true
    packagerOptions:
         scripts:
            - npm rebuild --arch=x64 --platform=linux --target=12.14.1 sharp

[webpack.config.js]

const path = require("path");
const slsw = require("serverless-webpack");
const nodeExternals = require("webpack-node-externals");

module.exports = {
  context: __dirname,
  mode: slsw.lib.webpack.isLocal ? "development" : "production",
  entry: slsw.lib.entries,
  devtool: slsw.lib.webpack.isLocal
    ? "cheap-module-eval-source-map"
    : "source-map",
  resolve: {
    extensions: [".mjs", ".json", ".ts"],
    symlinks: false,
    cacheWithContext: false,
  },
  output: {
    libraryTarget: "commonjs",
    path: path.join(__dirname, ".webpack"),
    filename: "[name].js",
  },
  target: "node",
  externals: [nodeExternals()],
  module: {
    rules: [
      // all files with a `.ts` or `.tsx` extension will be handled by `ts-loader`
      {
        test: /\.(tsx?)$/,
        loader: "ts-loader",
        exclude: [
          [
            path.resolve(__dirname, "node_modules"),
            path.resolve(__dirname, ".serverless"),
            path.resolve(__dirname, ".webpack"),
          ],
        ],
        options: {
          transpileOnly: true,
          experimentalWatchApi: true,
        },
      },
    ],
  },
  plugins: [
    // new ForkTsCheckerWebpackPlugin({
    //   eslint: true,
    //   eslintOptions: {
    //     cache: true
    //   }
    // })
  ],
};

I did change script command that commented previous examples.

But, didn't work.

[lovell]

Cross-posted at https://stackoverflow.com/questions/62039078/error-darwin-x64-binaries-cannot-be-used-on-the-linux-x64-platform-aws-lamb

[ghost]

Thank you..

I wrote that post..

[coreysnyder04]

So no luck still Seo? Are you also using layers when you upload/deploy your app? I wonder if that's the issue? Or at least related to it. I'm surprised others haven't faced this same problem.

[ghost]

I did change module ( sharp to jimp )

Jimp is working. ( zero native dependency )

But, Jimp slower than sharp..

Jimp : https://www.npmjs.com/package/jimp

sharp and jimp performance : https://sharp.pixelplumbing.com/performance

[coreysnyder04]

It seems I just got past this issue but updating my build layer to use:

npm install --arch=x64 --platform=linux sharp@0.25.0
find lib/node_modules/. -mtime +10950 -print -exec touch {} \; #2239  Reference: https://github.com/aws/aws-cli/issues/2639#issuecomment-391255985

After doing this it works.

[ghost]

Please, can you provide your serverless.yml ?

I don't want to use lambda layer.

[rickcha]

Same issue. It looks like serverless is rebuilding all packages ignoring my local node_modules while deploying. I also tried using docker to npm install sharp as linux, but no luck.

[ghost]

I solved this problem using aws codebuild.

Thank you.

https://stackoverflow.com/questions/62039078/error-darwin-x64-binaries-cannot-be-used-on-the-linux-x64-platform-aws-lamb/63224217#63224217

[ahmedmehanna]

I fixed this issue by forcing the webpack to run the scripts I am using serverless-bundle So the solution to fix it, it was the following ...

[serverless.yml]

plugins:
  - serverless-offline
  - serverless-pseudo-parameters
  - serverless-iam-roles-per-function
  - serverless-bundle

custom:
  bundle:
    packagerOptions:
      scripts:
        - rm -rf node_modules/sharp
        - npm install --arch=x64 --platform=linux sharp

https://sharp.pixelplumbing.com/install#aws-lambda https://www.npmjs.com/package/serverless-bundle

[bros4president]

I fixed this issue by forcing the webpack to run the scripts I am using serverless-bundle So the solution to fix it, it was the following ...

[serverless.yml]

plugins:
  - serverless-offline
  - serverless-pseudo-parameters
  - serverless-iam-roles-per-function
  - serverless-bundle

custom:
  bundle:
    packagerOptions:
      scripts:
        - rm -rf node_modules/sharp
        - npm install --arch=x64 --platform=linux sharp

https://sharp.pixelplumbing.com/install#aws-lambda https://www.npmjs.com/package/serverless-bundle

I run into the same problem this thread was describing, developing on Windows and deploying to lambda.

Your answer helped me a lot as I was using a similar stack of plugins. So thanks a lot.

One caveat is that rm -rf node_modules/sharp threw an error so I added only the next line to scripts and it worked.

I assume it overrode the Windows installation, but I’m not sure if it’s safe.

Thanks!

[hoonblizz]

@ahmedmehanna Thanks for the solution. Its works like charm.

[uniqueinx]

For those using esbuild plugin

custom: {
    esbuild: {
      bundle: true,
      minify: false,
      sourcemap: true,
      target: 'node14',
      define: { 'require.resolve': undefined },
      platform: 'node',
      concurrency: 10,
      external: ['sharp'],
      packagerOptions: {
        scripts: [
          'rm -rf node_modules/sharp',
          'npm install --arch=x64 --platform=linux sharp',
        ],
      },
    },
  },

[taojunnan]

I did change module ( sharp to jimp )

Jimp is working. ( zero native dependency )

But, Jimp slower than sharp..

Jimp : https://www.npmjs.com/package/jimp

sharp and jimp performance : https://sharp.pixelplumbing.com/performance

thanks!