aws-lambda-nodejs: NodejsFunction should provide a way to bundle local version of AWS SDK

[amliuyong]

Describe the bug

Lambda got below error when create EMR serverless application in NODEJS_18_X

2023-08-09T10:13:32.533Z    5147a21c-0512-41f7-89ed-213ca971bf3e    ERROR   Invoke Error    
{
    "errorType": "TypeError",
    "errorMessage": "Cannot read properties of undefined (reading 'X86_64')",
    "stack": [
        "TypeError: Cannot read properties of undefined (reading 'X86_64')",
        "    at createEMRServerlessApp (/var/task/index.js:2530:61)",
        "    at _handler (/var/task/index.js:2516:33)",
        "    at Runtime.handler (/var/task/index.js:2501:23)",
        "    at Runtime.handleOnceNonStreaming (file:///var/runtime/index.mjs:1083:29)"
    ]
}

Expected Behavior

lambda run without error

Current Behavior

lambda got error

Reproduction Steps

In lambda with below code to create EMR serverless application

import { EMRServerlessClient, CreateApplicationCommand, CreateApplicationCommandInput, DeleteApplicationCommand, Architecture } from '@aws-sdk/client-emr-serverless';

const input: CreateApplicationCommandInput = {
    name: props.name,
    releaseLabel: props.version,
    type: 'SPARK',
    architecture: Architecture.X86_64,
    networkConfiguration: {
      subnetIds: props.subnetIds.split(','),
      securityGroupIds: [props.secourityGroupId],
    },
    autoStartConfiguration: {
      enabled: true,
    },
    autoStopConfiguration: {
      enabled: true,
      idleTimeoutMinutes: parseInt(props.idleTimeoutMinutes),
    },
  };

 const command = new CreateApplicationCommand(input);
  const response = await emrServerlessClient.send(command);

Lambda was deployed by CDK code as below

  const fn = new NodejsFunction(scope, 'CreateEMRServelsssApplicationLambda', {
    runtime: Runtime.NODEJS_18_X,
    entry: join(
      __dirname,
      '..',
      'lambda',
      'emr-serverless-app',
      'index.ts',
    ),
    handler: 'handler',
    memorySize: 256,
    role,

   // more code not shown .... 

  });

Possible Solution

No response

Additional Information/Context

I tried below two workarounds, they all worked:

1. change lambda Runtime from NODEJS_18_X to NODEJS_16_X,
2. change the code in lambda from architecture: Architecture.X86_64, to architecture: 'X86_64',

CDK CLI Version

3.81.0

Framework Version

No response

Node.js Version

18

OS

Linux

Language

Typescript

Language Version

4.9.5

Other information

No response

[peterwoodworth]

If just changing the lambda to remove the Architecture enum works to unblock, then I'm pretty confused how that would be the case. It seems the error would just be with the Architecture import for some reason, but the v3 SDK does include this enum so it should be working if everything else is.

Can you reproduce this error when using Function construct instead of NodejsFunction?

[zxkane]

If just changing the lambda to remove the Architecture enum works to unblock, then I'm pretty confused how that would be the case. It seems the error would just be with the Architecture import for some reason, but the v3 SDK does include this enum so it should be working if everything else is.

Can you reproduce this error when using Function construct instead of NodejsFunction?

The Lmabda is written with Typescript, so using NodejsFunction to package it as final artifact running in Lambda. Do you mean moving to Function with our packaing script?

[peterwoodworth]

Ok on further review, to me it looks like Architecture was not supported on the default sdk version for lambdas created on Node 18 (3.188.0) . If I try to reference Architecture from this module on 3.188.0 locally, it does not exist as a type.

You will need to add a newer version of the SDK to the Lambda yourself in this case if the default does not provide what you need

[github-actions[bot]]

This issue has not received a response in a while. If you want to keep this issue open, please leave a comment below and auto-close will be canceled.

[zxkane]

It's another reason we use NodejsFunction to wrap the lambda function with Javascript. It can be built with the project's dependencies as a minified artifact. The project already uses the @aws-sdk/client-emr-serverless@3.388.0.

https://github.com/awslabs/clickstream-analytics-on-aws/blob/main/yarn.lock#L535-L536

[iliapolo]

@zxkane

It can be built with the project's dependencies as a minified artifact.

Yes, but not the SDK. The aws-sdk is marked as external and not bundled because we favor using the SDK available in the runtime instead. When specifying a Node18 runtime, we mark SDK v3 as external, and when specifying Node16 runtime, we mark SDK v2 as external. This is also why changing the runtime to Node16 works, even though your code uses SDK v3.

As another workaround, you can specify externalModules: [], which will bypass marking any package as external and will cause the bundler to bundle everything. We should probably provide a proper flag for this though, something like bundleSDK: true.

I'm going to mark this as a feature request.

[MrArnoldPalmer]

As of now, if you use the Runtime.NODEJS_LATEST version, it will not default to including the SDKs in the external modules. Additionally, if you need this runtime by default, you can use the @aws-cdk/aws-lambda-nodejs:useLatestRuntimeVersion feature flag. As @iliapolo pointed out, if you want to specify a specific runtime version and not exclude the sdks in bundling you can set externalModules: [].

We could potentially add another feature flag to not externalize sdks by default for any runtime as that feels like what the default should in fact be. @iliapolo thoughts?