'@smithy/node-http-handler' not included in lambda Node.js 20.x runtime

mellster2012 commented 3 weeks ago

Checkboxes for prior research

[X] I've gone through Developer Guide and API reference
[X] I've checked AWS Forums and StackOverflow.
[X] I've searched for previous similar issues and didn't find any solution.

Describe the bug

The documentation at https://github.com/aws/aws-sdk-js-v3/blob/main/supplemental-docs/CLIENTS.md#request-handler-requesthandler says to use '@smithy/node-http-handler' but the lambda fails with "errorMessage": "Cannot find package '@smithy/node-http-handler' imported from /var/task/index.mjs", "code": "ERR_MODULE_NOT_FOUND", regardless of whether the module is specified as a dependency in package.json or not.

SDK version number

@aws-sdk/client-s3 3.4x and 3.5x

Which JavaScript Runtime is this issue in?

Node.js

Details of the browser/Node.js/ReactNative version

Node.js v20

Reproduction Steps

import { GetObjectCommand, S3Client } from '@aws-sdk/client-s3';
import { NodeHttpHandler } from '@smithy/node-http-handler';

const s3 = new S3Client({
  requestHandler: new NodeHttpHandler({
    httpsAgent: new https.Agent({
      maxSockets: 150,
    }),
  })
});
[...]

Observed Behavior

2024-06-05T19:28:48.275Z    undefined   ERROR   Uncaught Exception  
{
    "errorType": "Error",
    "errorMessage": "Cannot find package '@smithy/node-http-handler' imported from /var/task/index.mjs",
    "code": "ERR_MODULE_NOT_FOUND",
    "stack": [
        "Error [ERR_MODULE_NOT_FOUND]: Cannot find package '@smithy/node-http-handler' imported from /var/task/index.mjs",
        "    at packageResolve (node:internal/modules/esm/resolve:858:9)",
        "    at moduleResolve (node:internal/modules/esm/resolve:931:18)",
        "    at moduleResolveWithNodePath (node:internal/modules/esm/resolve:1161:14)",
        "    at defaultResolve (node:internal/modules/esm/resolve:1204:79)",
        "    at ModuleLoader.defaultResolve (node:internal/modules/esm/loader:383:12)",
        "    at ModuleLoader.resolve (node:internal/modules/esm/loader:352:25)",
        "    at ModuleLoader.getModuleJob (node:internal/modules/esm/loader:227:38)",
        "    at ModuleWrap.<anonymous> (node:internal/modules/esm/module_job:87:39)",
        "    at link (node:internal/modules/esm/module_job:86:36)"
    ]
}

Expected Behavior

Module should be found.

Possible Solution

Workaround is to use import { NodeHttpHandler } from '@aws-sdk/node-http-handler'; instead.

Additional Information/Context

No response

aBurmeseDev commented 2 weeks ago

Hi @mellster2012 - thanks for reaching out.

Can you verify if you've installed smithy/node-http-handler in your packages? @aws-sdk/node-http-handler has been deprecated and some packages from @aws/sdk* were moved over to @smithy/*. Refer to this comment.

Smithy is a language for defining services and SDKs. It is open source software maintained by AWS.

Smithy is used to generate both the AWS SDK for JavaScript v3, and other similar SDKs not necessarily related to AWS. Therefore, we decided to detach the AWS SDK namespace from core components used outside of AWS, such as the generic node http handler.

aBurmeseDev commented 2 weeks ago

@mellster2012 - this's reproducible when running code in Lambda provided SDK and we were able to confirm it.

Please use the shortened syntax to configure requestHandler until fix is released. (ref: P135205927)

// Example: short form requestHandler configuration.
import { DynamoDBClient } from "@aws-sdk/client-dynamodb";

const client = new DynamoDBClient({
  requestHandler: {
    requestTimeout: 3_000,
    httpsAgent: { maxSockets: 25 },
  },
});

kuhe commented 2 weeks ago

The bundled SDK is provided as a convenience for developers building simpler functions or using the Lambda console for development

https://docs.aws.amazon.com/lambda/latest/operatorguide/sdks-functions.html

The AWS Lambda provided AWS SDK for JavaScript is only updated periodically, and may be missing features that the most recent AWS SDK version contains. When it does update, it is done automatically without a way to opt-out.

Therefore, in addition to the workaround shown above, for stability in production usage you should provide your own AWS SDK version as part of your uploaded application.

mellster2012 commented 2 weeks ago

The bundled SDK is provided as a convenience for developers building simpler functions or using the Lambda console for development

https://docs.aws.amazon.com/lambda/latest/operatorguide/sdks-functions.html

The AWS Lambda provided AWS SDK for JavaScript is only updated periodically, and may be missing features that the most recent AWS SDK version contains. When it does update, it is done automatically without a way to opt-out.

Therefore, in addition to the workaround shown above, for stability in production usage you should provide your own AWS SDK version as part of your uploaded application.

Providing the missing/additional dependency explicitly in package.json doesn't work either, though. Do you have to add an option to bundle your own AWS, such as mentioned here: https://github.com/aws/aws-cdk/issues/25492 ? And is there a way to instruct it to take the currently provided AWS SDK at runtime and add a few extra dependencies found in package.json, or do you have to either provide your (entire) own custom AWS SDK or nothing?

aBurmeseDev commented 2 weeks ago

Providing the missing/additional dependency explicitly in package.json doesn't work either, though

Can you elaborate more on that or share what you did with repro code? I've confirmed the workaround mentioned above with providing @smithy/node-http-handler dependency works. Let us know if we're missing something.

mellster2012 commented 2 weeks ago

Providing the missing/additional dependency explicitly in package.json doesn't work either, though

Can you elaborate more on that or share what you did with repro code? I've confirmed the workaround mentioned above with providing @smithy/node-http-handler dependency works. Let us know if we're missing something.

You are right, I missed that Code.fromAsset does not automatically bundle additional dependencies outside of the lambda handler folder. We can keep using @aws-sdk/node-http-handler (which is working fine) and wait until @smithy/node-http-handler is included or switch the build process for the affected lambda and include @smithy/node-http-handler as a dependency. Thank you!

aws / aws-sdk-js-v3