GEN2 runtime error: TypeError: __require.ensure is not a function

[anni1236012]

Environment information

System:
  OS: Linux 5.15 Ubuntu 22.04.4 LTS 22.04.4 LTS (Jammy Jellyfish)
  CPU: (12) x64 Intel(R) Core(TM) i7-9750H CPU @ 2.60GHz
  Memory: 8.87 GB / 15.49 GB
  Shell: /bin/bash
Binaries:
  Node: 20.11.1 - ~/.nvm/versions/node/v20.11.1/bin/node
  Yarn: 1.22.19 - ~/.yarn/bin/yarn
  npm: 10.8.0 - ~/.nvm/versions/node/v20.11.1/bin/npm
  pnpm: undefined - undefined
NPM Packages:
  @aws-amplify/backend: 1.0.3
  @aws-amplify/backend-cli: 1.0.4
  aws-amplify: 6.3.4
  aws-cdk: 2.144.0
  aws-cdk-lib: 2.144.0
  typescript: 5.4.5
AWS environment variables:
  AWS_STS_REGIONAL_ENDPOINTS = regional
  AWS_NODEJS_CONNECTION_REUSE_ENABLED = 1
  AWS_SDK_LOAD_CONFIG = 1
No CDK environment variables

Description

Looks like a build error. I've narrow down the issue to a pdf-parse package. Runtime error occurs at this step javascript const loader = new PDFLoader(pdfBlob);. Please see the complete code below.

langchain documentation for reference https://js.langchain.com/v0.2/docs/integrations/document_loaders/file_loaders/pdf/

Runtime Lambda Error

2024-06-08T03:23:36.085Z    daa44ece-01ec-4c9b-94cd-13003599a4cb    INFO    Error occured TypeError: __require.ensure is not a function
    at fakeWorkerFilesLoader (file:///var/task/index.mjs:947904:27)
    at setupFakeWorkerGlobal (file:///var/task/index.mjs:948571:17)
    at PDFWorker2.PDFWorker_setupFakeWorker [as _setupFakeWorker] (file:///var/task/index.mjs:948706:19)
    at PDFWorker2.PDFWorker_initialize [as _initialize] (file:///var/task/index.mjs:948698:24)
    at new PDFWorker2 (file:///var/task/index.mjs:948595:22)
    at getDocument (file:///var/task/index.mjs:947999:72)
    at PDFLoader2.parse (file:///var/task/index.mjs:962684:27)
    at async Runtime.handler (file:///var/task/index.mjs:970952:27)

handler.ts

import { Amplify } from "aws-amplify";
import AWS from "aws-sdk";
import { PDFLoader } from "@langchain/community/document_loaders/fs/pdf";
import { env } from "$amplify/env/CreateFileChatbotHandler";
import { AppSyncResolverHandler } from "aws-lambda";
const s3 = new AWS.S3();

Amplify.configure(
  {
    API: {
      GraphQL: {
        endpoint: env.AMPLIFY_DATA_GRAPHQL_ENDPOINT,
        region: env.AWS_REGION,
        defaultAuthMode: "identityPool",
      },
    },
  },
  {
    Auth: {
      credentialsProvider: {
        getCredentialsAndIdentityId: async () => ({
          credentials: {
            accessKeyId: env.AWS_ACCESS_KEY_ID,
            secretAccessKey: env.AWS_SECRET_ACCESS_KEY,
            sessionToken: env.AWS_SESSION_TOKEN,
          },
        }),
        clearCredentialsAndIdentityId: () => {
        },
      },
    },
  }
);

type InputArguments = {
  emailId?: string;
  identityId?: string;
  invokeType?: string;
  filePaths?: string[];
  content?: string;
};

export const handler: AppSyncResolverHandler<
  InputArguments,
  { emailId?: string; identityId?: string } | null | undefined
> = async (event) => {
  if (event?.arguments?.filePaths && event.arguments.filePaths.length > 0) {
    const filePath = event.arguments.filePaths[0];

    const data = await s3
      .getObject({
        Bucket: env.VOXAL_STORAGE_BUCKET_NAME,
        Key: filePath,
      })
      .promise();

    if (data?.Body) {
      let fileContent;
      console.log("type of buffer", typeof data.Body);
      console.log("contenmt", data.Body);

      if (Buffer.isBuffer(data.Body)) {
        const pdfBlob = new Blob([data.Body], { type: "application/pdf" });
        const loader = new PDFLoader(pdfBlob); <<==== Runtime error at this line
        fileContent = await loader.load();
      }
      return {
        content: fileContent,
        identityId: event.arguments.identityId,
      };
    }
  } 
};

package.json

{
  "type": "module",
  "dependencies": {
    "@langchain/community": "^0.2.9",
    "aws-sdk": "^2.1633.0",
    "langchain": "^0.2.5",
    "pdf-parse": "^1.1.1"
  },
  "devDependencies": {
    "@types/pdf-parse": "^1.1.4"
  }
}

tsconfig.json

{
  "compilerOptions": {
    "target": "es2022",
    "module": "es2022",
    "moduleResolution": "bundler",
    "resolveJsonModule": true,
    "esModuleInterop": true,
    "forceConsistentCasingInFileNames": true,
    "strict": true,
    "skipLibCheck": true,
    "paths": {
      "$amplify/*": ["../.amplify/generated/*"]
    }
  }
}

[kekami]

What repository structure do you have? Why aren't the dependencies specified in the root package.json?

[anni1236012]

package.json has the dependencies as shown above in the code and it is in the amplify directory. Here is the directory structure.

[kekami]

Why are they not defined in the root package.json? During build npm is being run from the root, and afaik npm does not automatically resolve nested package.json

Might to be the thing causing the issue, but that stood out to me.

[anni1236012]

root package.json is only for frontend build but amplify has it's own build process to build the backend and it will look for amplify/package.json

[kekami]

Not sure that is true, but perhaps it works none the less. I don't have any dependencies in mine, using a monorepo setup, and my lambdas are able to pull in any dependencies from my root package.json.

Could it perhaps be that the library requires some lambda layers or be incompatible with esbuild?

You could try deploying a NodeJsFunction construct to rule out amplify as the culprit.

[anni1236012]

I tried with NodeConstruct but still getting the same error, require.ensure is not a function. Below is my backend.ts . is the correct way of deploying NodeConstruct?

In GEN1, same package is working fine. I use require content in javascript and it has the node run 18. I don't know how it works in GEN1 because lambda is running in the same environment.

import { defineBackend, Backend } from "@aws-amplify/backend";
import { auth } from "./auth/resource";
import { data } from "./data/resource";
import path from "path";
import * as iam from "aws-cdk-lib/aws-iam";
import { NodejsFunction } from "aws-cdk-lib/aws-lambda-nodejs";
import * as lambda from "aws-cdk-lib/aws-lambda";
import { storage } from "./storage/resource";
import { Duration } from "aws-cdk-lib/core";
import { MyStack } from "./functions/CreateFilebasedChatbot/myStack";
import { CreateFileChatbotPermissions } from "./functions/CreateFilebasedChatbot/permissions";
import { CreateFileChatbotHandler } from "./functions/CreateFilebasedChatbot/resource";
import { fileURLToPath } from "url";

// ES module alternative to __dirname
const __filename = fileURLToPath(import.meta.url);
const __dirname = path.dirname(__filename);
/**
 * @see https://docs.amplify.aws/react/build-a-backend/ to add storage, functions, and more
 */
const backend = defineBackend({
  auth,
  data,
  storage,
});

// Create a new stack for the Lambda function
const lambdaStack = backend.createStack("LambdaStack");

// Define the Lambda function
const createFileChatbotLambda = new NodejsFunction(
  lambdaStack,
  "CreateFileChatbotLambda",
  {
    entry: path.join(
      __dirname,
      "./functions/CreateFilebasedChatbot/handler.ts"
    ),
    handler: "handler",
    runtime: lambda.Runtime.NODEJS_18_X,
    bundling: {
      externalModules: [
        "aws-sdk",
      ],
    },
  }
);

// Define the ARN for the function
const createFileChatbotLambdaARN = createFileChatbotLambda.functionArn;

// Create the IAM policy statement
const selfInvokePolicy = new iam.PolicyStatement({
  effect: iam.Effect.ALLOW,
  actions: ["lambda:InvokeFunction"],
  resources: [createFileChatbotLambdaARN],
});

// Create an inline policy and attach it to the Lambda role
const inlinePolicy = new iam.Policy(lambdaStack, "SelfInvokePolicy", {
  statements: [selfInvokePolicy],
});

if (createFileChatbotLambda.role) {
  createFileChatbotLambda.role.attachInlinePolicy(inlinePolicy);
}

// Grant self-invoke permissions
createFileChatbotLambda.addPermission("SelfInvokePermission", {
  principal: new iam.ServicePrincipal("lambda.amazonaws.com"),
  action: "lambda:InvokeFunction",
  sourceArn: createFileChatbotLambdaARN,
});

[edwardfoyle]

I haven't looked closely at this yet, but just wanted to clarify:

but amplify has it's own build process to build the backend and it will look for amplify/package.json

This isn't entirely true. The amplify/package.json file only tells Node to treat that folder as a module, it does not contain any information on dependencies. Dependencies should still be installed in the root package.json of the project.

[ykethan]

Hey @anni1236012, following up on this issue. Were you able to resolve this issue?

[ykethan]

Closing the issue due to inactivity. Do reach out to us if you are still experiencing this issue