Unable to run Sharp in AWS Amplify

[mchadwickchennault]

Question about an existing feature

When calling sharp within my Lambda function, I get the following error:

{
    "errorType": "Error",
    "errorMessage": "Could not load the \"sharp\" module using the linux-x64 runtime\nPossible solutions:\n- Ensure optional dependencies can be installed:\n    npm install --include=optional sharp\n- Ensure your package manager supports multi-platform installation:\n    See https://sharp.pixelplumbing.com/install#cross-platform\n- Add platform-specific dependencies:\n    npm install --os=linux --cpu=x64 sharp\n- Consult the installation documentation:\n    See https://sharp.pixelplumbing.com/install",
    "stack": [
        "Error: Could not load the \"sharp\" module using the linux-x64 runtime",
        "Possible solutions:",
        "- Ensure optional dependencies can be installed:",
        "    npm install --include=optional sharp",
        "- Ensure your package manager supports multi-platform installation:",
        "    See https://sharp.pixelplumbing.com/install#cross-platform",
        "- Add platform-specific dependencies:",
        "    npm install --os=linux --cpu=x64 sharp",
        "- Consult the installation documentation:",
        "    See https://sharp.pixelplumbing.com/install",
        "    at amplify/node_modules/sharp/lib/sharp.js (/var/task/index.js:2012:13)",
        "    at __require (/var/task/index.js:9:50)",
        "    at amplify/node_modules/sharp/lib/constructor.js (/var/task/index.js:2024:5)",
        "    at __require (/var/task/index.js:9:50)",
        "    at amplify/node_modules/sharp/lib/index.js (/var/task/index.js:6207:17)",
        "    at __require (/var/task/index.js:9:50)",
        "    at Object.<anonymous> (/var/task/index.js:11080:28)",
        "    at Module._compile (node:internal/modules/cjs/loader:1364:14)",
        "    at Module._extensions..js (node:internal/modules/cjs/loader:1422:10)",
        "    at Module.load (node:internal/modules/cjs/loader:1203:32)"
    ]
}

What are you trying to achieve?

I am attempting to run sharp in a Lambda function. The Lambda function is defined via CDK, but built by Amplify. The function targets Node 18.x. The image is "Amazon Linux 2023." I have attempted using both yarn and npm following the instructions found here: https://sharp.pixelplumbing.com/install#aws-lambda I have installed using npm install --cpu=x64 --os=linux --libc=glibc sharp and npm install --include=optional sharp

When you searched for similar issues, what did you find that might be related?

https://github.com/lovell/sharp/issues/4213

Please provide a minimal, standalone code sample, without other dependencies, that demonstrates this question

Please provide sample image(s) that help explain this question

[lovell]

https://sharp.pixelplumbing.com/install#aws-lambda

The node_modules directory of the deployment package must include binaries for either the linux-x64 or linux-arm64 platforms depending on the chosen architecture.

What are the contents of the node_modules/@img directory in the .zip file used as your deployment package?

Are you using any other framework? Is everything using the latest version? Are any other code bundlers being used? Please share as much information and configuration as possible.

[mchadwickchennault]

The contents of the @img directory are:

I am using Amplify V2 (the latest). My FE is bundled with Vite, but the BE uses the vanilla Amplify bundler. Under the hood, I believe this to be esbuild, but I am not sure it is possible specify settings for this.

[mchadwickchennault]

I made a little progress on this... I found I can set internal esbuild options with the following setup:

import * as url from 'node:url';
import { Runtime } from 'aws-cdk-lib/aws-lambda';
import * as lambda from 'aws-cdk-lib/aws-lambda-nodejs';
import { Construct } from 'constructs';
import { Duration } from 'aws-cdk-lib/core';

interface ImageProcessorProps {
  progressTopicArn: string;
  bucketName: string;
}

export class ImageProcessor extends Construct {
  public readonly function: lambda.NodejsFunction;

  constructor(scope: Construct, id: string, props: ImageProcessorProps) {
    super(scope, id);
    const { progressTopicArn } = props;
    this.function = new lambda.NodejsFunction(this, 'image-processor', {
      entry: url.fileURLToPath(new URL('./handler.ts', import.meta.url)),
      environment: {
        PROGRESS_TOPIC_ARN: progressTopicArn,
      },
      runtime: Runtime.NODEJS_18_X,
      timeout: Duration.minutes(15),
      memorySize: 1536,
      bundling: {
       // set esbuild args here
        esbuildArgs: {
          '--external:sharp': '',
        },
      },
    });
  }
}

However, now I am getting a "module not found" error.

[mchadwickchennault]

I found a solution. Hopefully it will help someone else in this situation. I created a package.json in the folder I defined my function in and installed only sharp. Then I added a bundle step to copy the node_modules folder to the output folder.

import * as url from 'node:url';
import { Runtime } from 'aws-cdk-lib/aws-lambda';
import * as lambda from 'aws-cdk-lib/aws-lambda-nodejs';
import { Construct } from 'constructs';
import { Duration } from 'aws-cdk-lib/core';

interface ImageProcessorProps {
  progressTopicArn: string;
  bucketName: string;
}

export class ImageProcessor extends Construct {
  public readonly function: lambda.NodejsFunction;

  constructor(scope: Construct, id: string, props: ImageProcessorProps) {
    super(scope, id);
    const { progressTopicArn } = props;
    this.function = new lambda.NodejsFunction(this, 'image-processor', {
      entry: url.fileURLToPath(new URL('./handler.ts', import.meta.url)),
      environment: {
        PROGRESS_TOPIC_ARN: progressTopicArn,
      },
      runtime: Runtime.NODEJS_18_X,
      timeout: Duration.minutes(15),
      memorySize: 1536,
      // these are the essential settings to get it to work
      bundling: {
        esbuildArgs: {
          '--external:sharp': '',
        },
        commandHooks: {
          beforeBundling() {
            return [];
          },
          afterBundling(inputDir: string, outputDir: string) {
            return [
              `echo "Copying node_modules from ${inputDir}"`,
              `cp -r ${inputDir}/amplify/custom/functions/imageProcessor/node_modules ${outputDir}/node_modules/`,
            ];
          },
          beforeInstall() {
            return [];
          },
        },
      },
    });
  }
}

[lovell]

The CDK BundlingOptions suggest you might also be able to use the nodeModules property for this, something like the following (untested):

bundling: {
  nodeModules: ["sharp"]
}

[mchadwickchennault]

I tried that and alas it did not work. It basically automatically does what I have done by manually copying the node_module folder. However, the build breaks when you add that property.