Clients are broken in Turbopack

AllanZhengYP commented 1 year ago

Checkboxes for prior research

[X] I've gone through Developer Guide and API reference
[X] I've checked AWS Forums and StackOverflow.
[X] I've searched for previous similar issues and didn't find any solution.

Describe the bug

The AWS SDK packages to broken in Turbopack, complaining node-specific artifacts cannot be resolved. This is not a strict bug report since Turbopack hasn't been stablized yet. However, it's the minor issues in the AWS SDK blocking the Turbopack.

I can implement the fix if I have bendwidth. I documented the detailed changes needed below, feel free to pick them up.

SDK version number

@aws-sdk/client-*@all

Which JavaScript Runtime is this issue in?

Browser

Details of the browser/Node.js/ReactNative version

N/A

Reproduction Steps

Checkout the with-s3-client branch of the Next 13 test repo, and start dev server with yarn dev.

Observed Behavior

FS related module cannot be resolved error.

Expected Behavior

The bundling should not have problems

Possible Solution

Because of the lack of the documentation I spent quite a lot of time in the Turbopack bundling and found the 3 fixed below is needed for the code to work in Turbopack for client-side rendering:

Add extensions to browser field: Latest version of AWS SDK browser field looks like:
```
"browser": {
    "./dist-es/runtimeConfig": "./dist-es/runtimeConfig.browser"
  },
```
It works well with all the other bundlers, but Turbopack is more strict. It requires .js extension for both key and value.
The package @aws-sdk/util-user-agent-node specifies optional dependency "aws-crt". The dependency is, even though only exists in Node.js specific code path, still loaded and analyzed by Turbopack and hence causing "Unable to resolve package" error. This particular issue may be a Turbopack issue, but I find it can be worked around with a fix on AWS SDK side — using createRequire in the dist-es artifact to require optional dependencies instead of calling require() directly. The fix on the AWS SDK side is more "correct" as the current implementation does miss some considerations.
The package @aws-sdk/signature-v4-multi-region packaga contains require() api to assert the optional Node.js specific dependency @aws-sdk/signature-v4-crt (link). The being in the browser code path is causing errors in the Turbopack. This code can be moved to a separate file and disabled in Node.js runtime:
```
// loadSigv4aSignerCtor.js
export const loadSignv4aSignerCtr = typeof require === "function" && require("@aws-sdk/signature-v4-crt").CrtSignerV4;
```

// SignatureV4MultiRegion.js getSigv4aSigner() { if (!this.sigv4aSigner) { let CrtSignerV4; try { CrtSignerV4 = loadSignv4aSignerCtr(); // <==== here's the change if (typeof CrtSignerV4 !== "function") throw new Error(); } catch (e) { e.message = ${e.message}\nPlease check if you have installed "@aws-sdk/signature-v4-crt" package explicitly. \n + "For more information please go to " + "https://github.com/aws/aws-sdk-js-v3#functionality-requiring-aws-common-runtime-crt"; throw e; } this.sigv4aSigner = new CrtSignerV4({ ...this.signerOptions, signingAlgorithm: 1, }); } return this.sigv4aSigner; }

//package.json "browser": { "./dist-es/loadSigv4aSignerCtor.js": false, "@aws-sdk/signature-v4-crt": false },



### Additional Information/Context

_No response_

koshic commented 1 year ago

It would be nice to use exports with conditions as it described in https://nodejs.org/dist/latest-v19.x/docs/api/packages.html#exports without using legacy fields like 'browser', 'module', etc.

yenfryherrerafeliz commented 1 year ago

Hi @AllanZhengYP, thanks for opening this issue. I was able to reproduce the reported behavior by doing the steps below. So, I will mark this issue as for review so we can address this further.

Steps to reproduce:

Cloned the following repo
From the root directory of the project I ran yarn dev
Accessed from the web browser to " http://localhost:3000"
I got the errors below:

```console error - [resolve] node_modules/@aws-sdk/signature-v4-multi-region/dist-es/SignatureV4MultiRegion.js Error resolving commonjs request unable to resolve module "@aws-sdk/signature-v4-crt" node_modules/aws-crt/dist/native/binding.js Error resolving commonjs request unable to resolve dynamic warning - [analyze] node_modules/@aws-sdk/client-s3/dist-es/pagination/Interfaces.js unexpected export * export * used with module [project-with-next]/node_modules/@aws-sdk/client-s3/dist-es/pagination/Interfaces.js which has no exports Typescript only: Did you want to import only types with `export type * from "..."`? node_modules/@aws-sdk/client-sso/dist-es/pagination/Interfaces.js unexpected export * export * used with module [project-with-next]/node_modules/@aws-sdk/client-sso/dist-es/pagination/Interfaces.js which has no exports Typescript only: Did you want to import only types with `export type * from "..."`? node_modules/@aws-sdk/config-resolver/dist-es/regionInfo/PartitionHash.js unexpected export * export * used with module [project-with-next]/node_modules/@aws-sdk/config-resolver/dist-es/regionInfo/PartitionHash.js which has no exports Typescript only: Did you want to import only types with `export type * from "..."`? node_modules/@aws-sdk/config-resolver/dist-es/regionInfo/RegionHash.js unexpected export * export * used with module [project-with-next]/node_modules/@aws-sdk/config-resolver/dist-es/regionInfo/RegionHash.js which has no exports Typescript only: Did you want to import only types with `export type * from "..."`? node_modules/@aws-sdk/credential-provider-imds/dist-es/types.js unexpected export * export * used with module [project-with-next]/node_modules/@aws-sdk/credential-provider-imds/dist-es/types.js which has no exports Typescript only: Did you want to import only types with `export type * from "..."`? node_modules/@aws-sdk/credential-provider-sso/dist-es/types.js unexpected export * export * used with module [project-with-next]/node_modules/@aws-sdk/credential-provider-sso/dist-es/types.js which has no exports Typescript only: Did you want to import only types with `export type * from "..."`? node_modules/@aws-sdk/eventstream-codec/dist-es/Message.js unexpected export * export * used with module [project-with-next]/node_modules/@aws-sdk/eventstream-codec/dist-es/Message.js which has no exports Typescript only: Did you want to import only types with `export type * from "..."`? node_modules/@aws-sdk/middleware-endpoint/dist-es/types.js unexpected export * export * used with module [project-with-next]/node_modules/@aws-sdk/middleware-endpoint/dist-es/types.js which has no exports Typescript only: Did you want to import only types with `export type * from "..."`? node_modules/@aws-sdk/middleware-retry/dist-es/types.js unexpected export * export * used with module [project-with-next]/node_modules/@aws-sdk/middleware-retry/dist-es/types.js which has no exports Typescript only: Did you want to import only types with `export type * from "..."`? node_modules/@aws-sdk/protocol-http/dist-es/httpHandler.js unexpected export * export * used with module [project-with-next]/node_modules/@aws-sdk/protocol-http/dist-es/httpHandler.js which has no exports Typescript only: Did you want to import only types with `export type * from "..."`? node_modules/@aws-sdk/shared-ini-file-loader/dist-es/types.js unexpected export * export * used with module [project-with-next]/node_modules/@aws-sdk/shared-ini-file-loader/dist-es/types.js which has no exports Typescript only: Did you want to import only types with `export type * from "..."`? node_modules/@aws-sdk/types/dist-es/abort.js unexpected export * export * used with module [project-with-next]/node_modules/@aws-sdk/types/dist-es/abort.js which has no exports Typescript only: Did you want to import only types with `export type * from "..."`? node_modules/@aws-sdk/types/dist-es/auth.js unexpected export * export * used with module [project-with-next]/node_modules/@aws-sdk/types/dist-es/auth.js which has no exports Typescript only: Did you want to import only types with `export type * from "..."`? node_modules/@aws-sdk/types/dist-es/client.js unexpected export * export * used with module [project-with-next]/node_modules/@aws-sdk/types/dist-es/client.js which has no exports Typescript only: Did you want to import only types with `export type * from "..."`? node_modules/@aws-sdk/types/dist-es/command.js unexpected export * export * used with module [project-with-next]/node_modules/@aws-sdk/types/dist-es/command.js which has no exports Typescript only: Did you want to import only types with `export type * from "..."`? node_modules/@aws-sdk/types/dist-es/credentials.js unexpected export * export * used with module [project-with-next]/node_modules/@aws-sdk/types/dist-es/credentials.js which has no exports Typescript only: Did you want to import only types with `export type * from "..."`? node_modules/@aws-sdk/types/dist-es/crypto.js unexpected export * export * used with module [project-with-next]/node_modules/@aws-sdk/types/dist-es/crypto.js which has no exports Typescript only: Did you want to import only types with `export type * from "..."`? node_modules/@aws-sdk/types/dist-es/eventStream.js unexpected export * export * used with module [project-with-next]/node_modules/@aws-sdk/types/dist-es/eventStream.js which has no exports Typescript only: Did you want to import only types with `export type * from "..."`? node_modules/@aws-sdk/types/dist-es/http.js unexpected export * export * used with module [project-with-next]/node_modules/@aws-sdk/types/dist-es/http.js which has no exports Typescript only: Did you want to import only types with `export type * from "..."`? node_modules/@aws-sdk/types/dist-es/logger.js unexpected export * export * used with module [project-with-next]/node_modules/@aws-sdk/types/dist-es/logger.js which has no exports Typescript only: Did you want to import only types with `export type * from "..."`? node_modules/@aws-sdk/types/dist-es/middleware.js unexpected export * export * used with module [project-with-next]/node_modules/@aws-sdk/types/dist-es/middleware.js which has no exports Typescript only: Did you want to import only types with `export type * from "..."`? node_modules/@aws-sdk/types/dist-es/pagination.js unexpected export * export * used with module [project-with-next]/node_modules/@aws-sdk/types/dist-es/pagination.js which has no exports Typescript only: Did you want to import only types with `export type * from "..."`? node_modules/@aws-sdk/types/dist-es/profile.js unexpected export * export * used with module [project-with-next]/node_modules/@aws-sdk/types/dist-es/profile.js which has no exports Typescript only: Did you want to import only types with `export type * from "..."`? node_modules/@aws-sdk/types/dist-es/response.js unexpected export * export * used with module [project-with-next]/node_modules/@aws-sdk/types/dist-es/response.js which has no exports Typescript only: Did you want to import only types with `export type * from "..."`? node_modules/@aws-sdk/types/dist-es/serde.js unexpected export * export * used with module [project-with-next]/node_modules/@aws-sdk/types/dist-es/serde.js which has no exports Typescript only: Did you want to import only types with `export type * from "..."`? node_modules/@aws-sdk/types/dist-es/shapes.js unexpected export * export * used with module [project-with-next]/node_modules/@aws-sdk/types/dist-es/shapes.js which has no exports Typescript only: Did you want to import only types with `export type * from "..."`? node_modules/@aws-sdk/types/dist-es/signature.js unexpected export * export * used with module [project-with-next]/node_modules/@aws-sdk/types/dist-es/signature.js which has no exports Typescript only: Did you want to import only types with `export type * from "..."`? node_modules/@aws-sdk/types/dist-es/stream.js unexpected export * export * used with module [project-with-next]/node_modules/@aws-sdk/types/dist-es/stream.js which has no exports Typescript only: Did you want to import only types with `export type * from "..."`? node_modules/@aws-sdk/types/dist-es/token.js unexpected export * export * used with module [project-with-next]/node_modules/@aws-sdk/types/dist-es/token.js which has no exports Typescript only: Did you want to import only types with `export type * from "..."`? node_modules/@aws-sdk/types/dist-es/transfer.js unexpected export * export * used with module [project-with-next]/node_modules/@aws-sdk/types/dist-es/transfer.js which has no exports Typescript only: Did you want to import only types with `export type * from "..."`? node_modules/@aws-sdk/types/dist-es/util.js unexpected export * export * used with module [project-with-next]/node_modules/@aws-sdk/types/dist-es/util.js which has no exports Typescript only: Did you want to import only types with `export type * from "..."`? node_modules/@aws-sdk/types/dist-es/waiter.js unexpected export * export * used with module [project-with-next]/node_modules/@aws-sdk/types/dist-es/waiter.js which has no exports Typescript only: Did you want to import only types with `export type * from "..."`? node_modules/@aws-sdk/util-endpoints/dist-es/types/EndpointRuleObject.js unexpected export * export * used with module [project-with-next]/node_modules/@aws-sdk/util-endpoints/dist-es/types/EndpointRuleObject.js which has no exports Typescript only: Did you want to import only types with `export type * from "..."`? node_modules/@aws-sdk/util-endpoints/dist-es/types/ErrorRuleObject.js unexpected export * export * used with module [project-with-next]/node_modules/@aws-sdk/util-endpoints/dist-es/types/ErrorRuleObject.js which has no exports Typescript only: Did you want to import only types with `export type * from "..."`? node_modules/@aws-sdk/util-endpoints/dist-es/types/RuleSetObject.js unexpected export * export * used with module [project-with-next]/node_modules/@aws-sdk/util-endpoints/dist-es/types/RuleSetObject.js which has no exports Typescript only: Did you want to import only types with `export type * from "..."`? node_modules/@aws-sdk/util-endpoints/dist-es/types/TreeRuleObject.js unexpected export * export * used with module [project-with-next]/node_modules/@aws-sdk/util-endpoints/dist-es/types/TreeRuleObject.js which has no exports Typescript only: Did you want to import only types with `export type * from "..."`? node_modules/@aws-sdk/util-endpoints/dist-es/types/shared.js unexpected export * export * used with module [project-with-next]/node_modules/@aws-sdk/util-endpoints/dist-es/types/shared.js which has no exports Typescript only: Did you want to import only types with `export type * from "..."`? [parse] node_modules/@aws-sdk/credential-provider-web-identity/dist-es/fromTokenFile.js node_modules/@aws-sdk/credential-provider-web-identity/dist-es/fromTokenFile.js:19:26 lint TP1004 fs.readFileSync((???*0* | ???*1*), {"encoding": "ascii"}) is very dynamic 15 throw new CredentialsProviderError("Web identity configuration not specified"); 16 } 17 return fromWebToken({ 18 ...init, 19 > webIdentityToken: readFileSync(webIdentityTokenFile, { encoding: "ascii" }), 20 roleArn, 21 roleSessionName, 22 })(); 23 }; - *0* unsupported expression - *1* ???*2*["AWS_WEB_IDENTITY_TOKEN_FILE"] ⚠️ property on unknown - *2* process*3*["env"] ⚠️ unsupported property on Node.js process object - *3* process: The Node.js process module: https://nodejs.org/api/process.html node_modules/@aws-sdk/hash-stream-node/dist-es/fileStreamHasher.js node_modules/@aws-sdk/hash-stream-node/dist-es/fileStreamHasher.js:8:26 lint TP1004 fs.createReadStream(???*0*, {"start": ???*2*, "end": ???*4*}) is very dynamic 4 if (!isReadStream(fileStream)) { 5 reject(new Error("Unable to calculate hash for non-file streams.")); 6 return; 7 } 8 > const fileStreamTee = createReadStream(fileStream.path, { 9 start: fileStream.start, 10 end: fileStream.end, 11 }); 12 const hash = new hashCtor(); - *0* ???*1*["path"] ⚠️ property on unknown - *1* fileStream ⚠️ pattern without value - *2* ???*3*["start"] ⚠️ property on unknown - *3* fileStream ⚠️ pattern without value - *4* ???*5*["end"] ⚠️ property on unknown - *5* fileStream ⚠️ pattern without value node_modules/aws-crt/dist/native/binding.js node_modules/aws-crt/dist/native/binding.js:59:18 lint TP1002 require(???*0*) is very dynamic 55 ]; 56 let binding; 57 for (const path of search_paths) { 58 if ((0, fs_1.existsSync)(path + '.node')) { 59 > binding = require(path); 60 break; 61 } 62 } 63 if (binding == undefined) { - *0* path ⚠️ pattern without value event - updated in 2042ms error - [resolve] node_modules/@aws-sdk/credential-provider-process/dist-es/resolveProcessCredentials.js Error resolving esm request unable to resolve module "child_process" node_modules/@aws-sdk/credential-provider-web-identity/dist-es/fromTokenFile.js Error resolving esm request unable to resolve module "fs" node_modules/@aws-sdk/hash-stream-node/dist-es/fileStreamHasher.js Error resolving esm request unable to resolve module "fs" node_modules/@aws-sdk/node-http-handler/dist-es/node-http2-handler.js Error resolving esm request unable to resolve module "http2" node_modules/@aws-sdk/shared-ini-file-loader/dist-es/getSSOTokenFromFile.js Error resolving esm request unable to resolve module "fs" node_modules/@aws-sdk/shared-ini-file-loader/dist-es/slurpFile.js Error resolving esm request unable to resolve module "fs" node_modules/@aws-sdk/util-body-length-node/dist-es/calculateBodyLength.js Error resolving esm request unable to resolve module "fs" node_modules/mqtt/lib/connect/tcp.js Error resolving commonjs request unable to resolve module "net" node_modules/mqtt/lib/connect/tls.js Error resolving commonjs request unable to resolve module "net" Error resolving commonjs request unable to resolve module "tls" node_modules/next/dist/compiled/browserify-zlib/index.js Error resolving commonjs request unable to resolve module "assert" node_modules/ws/lib/buffer-util.js Error resolving commonjs request unable to resolve module "bufferutil" node_modules/ws/lib/sender.js Error resolving commonjs request unable to resolve module "net" Error resolving commonjs request unable to resolve module "tls" node_modules/ws/lib/validation.js Error resolving commonjs request unable to resolve module "utf-8-validate" node_modules/ws/lib/websocket-server.js Error resolving commonjs request unable to resolve module "net" Error resolving commonjs request unable to resolve module "tls" node_modules/ws/lib/websocket.js Error resolving commonjs request unable to resolve module "net" Error resolving commonjs request unable to resolve module "tls" ```

Thanks!

Tomekmularczyk commented 1 year ago

Hi, any update on this? I get this problem with NextJs 13 when using webpack. The only difference is that I have app directory enabled (https://github.com/aws-amplify/amplify-js/issues/11030). It makes working with the project problematic as it slows down bundling a lot.

Module not found: Can't resolve 'aws-crt' in '/Users/tomekm/Desktop/tixy-frontend/node_modules/@aws-sdk/client-lex-runtime-service/node_modules/@aws-sdk/util-user-agent-node/dist-cjs'

Import trace for requested module:
./node_modules/@aws-sdk/client-lex-runtime-service/node_modules/@aws-sdk/util-user-agent-node/dist-cjs/is-crt-available.js
./node_modules/@aws-sdk/client-lex-runtime-service/node_modules/@aws-sdk/util-user-agent-node/dist-cjs/index.js
./node_modules/@aws-sdk/client-lex-runtime-service/dist-cjs/runtimeConfig.js
./node_modules/@aws-sdk/client-lex-runtime-service/dist-cjs/LexRuntimeServiceClient.js
./node_modules/@aws-sdk/client-lex-runtime-service/dist-cjs/index.js
./node_modules/@aws-amplify/interactions/lib/Providers/AWSLexProvider.js
./node_modules/@aws-amplify/interactions/lib/index.js
./node_modules/aws-amplify/lib/index.js
./src/graphql/amplifyConfigure.ts

./node_modules/@aws-sdk/client-lex-runtime-v2/node_modules/@aws-sdk/util-user-agent-node/dist-cjs/is-crt-available.js
Module not found: Can't resolve 'aws-crt' in '/Users/

ndueber commented 1 year ago

Is there an ETA on the fix? As of NextJs 13.4, the app directory was marked stable. This error is inhibiting development with a popular and widely used framework.

RanVaknin commented 1 year ago

Like @AllanZhengYP mentioned:

This is not a strict bug report since Turbopack hasn't been stablized yet. However, it's the minor issues in the AWS SDK blocking the Turbopack.

We will look into it when Turbopack becomes stable.

@ndueber and @Tomekmularczyk the issue you are describing is unrelated to Turbopack. Please open a separate issue, and fill the intake form and repro steps so we can better assist you.

Thanks, Ran~

Tomekmularczyk commented 1 year ago

Like @AllanZhengYP mentioned:

This is not a strict bug report since Turbopack hasn't been stablized yet. However, it's the minor issues in the AWS SDK blocking the Turbopack.

We will look into it when Turbopack becomes stable.

@ndueber and @Tomekmularczyk the issue you are describing is unrelated to Turbopack. Please open a separate issue, and fill the intake form and repro steps so we can better assist you.

Thanks, Ran~

There is an issue opened already: https://github.com/aws-amplify/amplify-js/issues/11030

davecarlson commented 1 year ago

Like @AllanZhengYP mentioned:

This is not a strict bug report since Turbopack hasn't been stablized yet. However, it's the minor issues in the AWS SDK blocking the Turbopack.

We will look into it when Turbopack becomes stable.

@ndueber and @Tomekmularczyk the issue you are describing is unrelated to Turbopack. Please open a separate issue, and fill the intake form and repro steps so we can better assist you.

Thanks, Ran~

@RanVaknin - This issue still happens if you are using webpack instead of turbopack with next's app directory, so it does not make sense to wait until turbopack is "stable" before fixing.

snowden-fu commented 1 year ago

I want. to say this problem still exists when i am using SES service

- wait compiling /api/feedback/route (client and server)...
- warn ./node_modules/@aws-sdk/util-user-agent-node/dist-cjs/is-crt-available.js
Module not found: Can't resolve 'aws-crt' in '/Users/snowden/Documents/GitHub/cv-cover/node_modules/@aws-sdk/util-user-agent-node/dist-cjs'

Import trace for requested module:
./node_modules/@aws-sdk/util-user-agent-node/dist-cjs/is-crt-available.js
./node_modules/@aws-sdk/util-user-agent-node/dist-cjs/index.js
./node_modules/@aws-sdk/client-sesv2/dist-cjs/runtimeConfig.js
./node_modules/@aws-sdk/client-sesv2/dist-cjs/SESv2Client.js
./node_modules/@aws-sdk/client-sesv2/dist-cjs/index.js
./src/app/api/feedback/route.ts
Failed to send email.

Rieranthony commented 1 year ago

Any news on that? This is blocking a lot of NextJS 13 users. Thanks! :)

JoshElias commented 12 months ago

+1

kevoj7 commented 11 months ago

+1

Luferov commented 10 months ago

+1

oliverjohns commented 9 months ago

+1

joaopedrodcf commented 8 months ago

Hey guys I also had this problem when using v5 of the lib, but after upgrading to aws-amplify v6 turbopack works without errors 💯

Hope this help others 💯

regexyl commented 3 months ago

@joaopedrodcf Hey guys I also had this problem when using v5 of the lib, but after upgrading to aws-amplify v6 turbopack works without errors 💯

Same here, I've done the following upgrades and this problem has gone away:

Before

    "@aws-sdk/client-api-gateway": "^3.370.0",
    "@aws-sdk/client-lambda": "^3.548.0",
    "@aws-sdk/client-s3": "^3.332.0",
    "@aws-sdk/client-sfn": "^3.319.0",
    "@aws-sdk/client-sqs": "^3.328.0",
    "@aws-sdk/credential-provider-env": "^3.535.0",
    "@aws-sdk/s3-request-presigner": "^3.332.0",

After

    "@aws-sdk/client-api-gateway": "^3.583.0",
    "@aws-sdk/client-lambda": "^3.583.0",
    "@aws-sdk/client-s3": "^3.583.0",
    "@aws-sdk/client-sfn": "^3.583.0",
    "@aws-sdk/client-sqs": "^3.583.0",
    "@aws-sdk/s3-request-presigner": "^3.583.0"

aws / aws-sdk-js-v3