search

aws / aws-sdk-js-v3

Modularized AWS SDK for JavaScript.
Apache License 2.0
3.12k stars 578 forks source link

Module not found: Can't resolve 'fs' in 'node_modules/@aws-sdk/lib-storage/dist/es #2564

Closed fanatic75 closed 1 year ago

fanatic75 commented 3 years ago

Describe the bug

Angular project not building because of FS error, is there no way to upload big files to s3 via Angular 11 with v3 sdk ?

Your environment

Angular - 11, Win 10

SDK version number

3.20.0

Is the issue in the browser/Node.js/ReactNative?

Browser

Details of the browser/Node.js/ReactNative version

Paste output of npx envinfo --browsers or node -v or react-native -v v14.15.1

Steps to reproduce

 const uploadParams = {
      Bucket: this.bucketName,
      Key: `podcasts/${this.userInfo.identityId}/${file.name}`,
      Body: file
    };
     try {
      const parallelUploads3 = new Upload({
        client: this.s3,
        params: uploadParams,
      });

      parallelUploads3.on("httpUploadProgress",progressFunc);

      await parallelUploads3.done();

    } catch (err) {
      return console.error(err);
    }

Observed behavior

Angular app crashes

Expected behavior

Use the library

Screenshots

If applicable, add screenshots to help explain your problem.

Additional context

duplicate of #2136

grothem commented 3 years ago

I was able to get it working by using this fix cross posted in webpack issues. I had to use the @angular-builders/custom-webpack package to make it work.

grothem commented 3 years ago

ahh scratch that. The build succeeds, but now it's failing at run time 😕

fanatic75 commented 3 years ago

For now I have moved to v2 till this is sorted out.

ajredniwja commented 3 years ago

Hi @fanatic75 thanks for bringing this one, I wasn't able to reproduce it. This was initially because fs is not available in browser and the SDK tries to load it, but I made a fix to that. I only see a warning but no errors. Can you setup a repo with your example that duplicates the error?

grothem commented 3 years ago

@ajredniwja Here's a repo that reproduces the issue. I used Angular CLI to generate a new v11 project, then added @aws-sdk/lib-storage and @aws-sdk/client-s3, then just pasted in the example from the lib-storage repo.

ajredniwja commented 3 years ago

@grothem thanks, I'll try to work with it, would be some time.

ajredniwja commented 3 years ago

Hi there, it turns out to be a dependency issue just add: 

  "browser": {
    "crypto": false,
    "fs": false,
    "path": false,
    "os": false,
    "net": false,
    "stream": false,
    "tls": false
  }

In your package.json and it should work as expected.

fanatic75 commented 3 years ago

Hey @ajredniwja , I added this object in package.json but didn't work. Thanks

ajredniwja commented 3 years ago

@fanatic75, let me share my package.json

ajredniwja commented 3 years ago 
{
  "name": "aws-lib-storage",
  "version": "0.0.0",
  "scripts": {
    "ng": "ng",
    "start": "ng serve",
    "build": "ng build",
    "test": "ng test",
    "lint": "ng lint",
    "e2e": "ng e2e"
  },
  "browser": {
    "crypto": false,
    "fs": false,
    "path": false,
    "os": false,
    "net": false,
    "stream": false,
    "tls": false
  },
  "private": true,

  "dependencies": {
    "@angular/animations": "~11.2.14",
    "@angular/common": "~11.2.14",
    "@angular/compiler": "~11.2.14",
    "@angular/core": "~11.2.14",
    "@angular/forms": "~11.2.14",
    "@angular/platform-browser": "~11.2.14",
    "@angular/platform-browser-dynamic": "~11.2.14",
    "@angular/router": "~11.2.14",
    "ng": "0.0.0",
    "rxjs": "~6.6.0",
    "tslib": "^2.0.0",
    "zone.js": "~0.11.3",
    "@aws-sdk/client-cognito-identity": "^3.15.0",
    "@aws-sdk/client-lex-runtime-service": "^3.15.0",
    "@aws-sdk/client-s3": "^3.21.0",
    "@aws-sdk/client-sqs": "^3.21.0",
    "@aws-sdk/credential-provider-cognito-identity": "^3.7.0",
    "@aws-sdk/lib-storage": "3.21.0",
    "@aws-sdk/s3-request-presigner": "^3.16.0",
    "@testing-library/jest-dom": "^5.11.4",
    "@testing-library/user-event": "^12.1.10",
    "axios": "^0.21.1",
    "fs": "0.0.1-security",
    "web-vitals": "^1.0.1"
  },
  "devDependencies": {
    "@angular-devkit/build-angular": "~0.1102.13",
    "@angular/cli": "~11.2.14",
    "@angular/compiler-cli": "~11.2.14",
    "@types/jasmine": "~3.6.0",
    "@types/node": "^12.11.1",
    "codelyzer": "^6.0.0",
    "jasmine-core": "~3.6.0",
    "jasmine-spec-reporter": "~5.0.0",
    "karma": "~6.1.0",
    "karma-chrome-launcher": "~3.1.0",
    "karma-coverage": "~2.0.3",
    "karma-jasmine": "~4.0.0",
    "karma-jasmine-html-reporter": "~1.5.0",
    "protractor": "~7.0.0",
    "ts-node": "~8.3.0",
    "tslint": "~6.1.0",
    "typescript": "~4.1.5"
  }
}

This is what I used, some of which was copied from the react project I had and the angular app ran perfectly uploading stuff on S3.

grothem commented 3 years ago

Thanks @ajredniwja, that did fix the build errors for me. Is this something that could be added to the @aws-sdk/lib-storage module package.json, instead of having the end users specify this config?

ajredniwja commented 3 years ago

@grothem yes, would need to find which package it is.

ignlg commented 3 years ago

After applying those fixes, it still fails with Angular 12, that uses Webpack v5.

This is the core of the package.json used:

{
  ...
  "dependencies": {
    "@angular/animations": "^12.1.3",
    "@angular/common": "^12.1.3",
    "@angular/compiler": "^12.1.3",
    "@angular/core": "^12.1.3",
    "@angular/forms": "^12.1.3",
    "@angular/localize": "^12.1.3",
    "@angular/platform-browser": "^12.1.3",
    "@angular/platform-browser-dynamic": "^12.1.3",
    "@angular/router": "^12.1.3",
    "@aws-sdk/client-s3": "^3.22.0",
    "@aws-sdk/lib-storage": "^3.23.0",
    "fs": "^0.0.1-security",
    "zone.js": "^0.11.3"
  },
  "devDependencies": {
    "@angular-devkit/build-angular": "~12.1.3",
    "@angular-devkit/core": "12.1.3",
    "@angular/cli": "^12.1.3",
    "@angular/compiler-cli": "^12.1.3",
    "@angular/language-service": "^12.1.3",
    "typescript": "4.3.5"
  },
  "browser": {
    "crypto": false,
    "fs": false,
    "net": false,
    "os": false,
    "path": false,
    "stream": false,
    "tls": false
  },
  ...
}

Also tsconfig is configured with:

    "lib": ["ES2020", "ES2019", "ESNext", "DOM"],

Any ideas?

brett-vendia commented 3 years ago

Failing in a React app using Next 11 (which is also Webpack 5), even after adding the "browser" entry

grothem commented 3 years ago

@ajredniwja just wondering if there was any update on this. I'm now running into issues using v2 of aws-sdk and angular 12 (see this issue)

ajredniwja commented 3 years ago

testing this now, working in react app using the latest packages, the test included uploading some stuff to S3

ajredniwja commented 3 years ago

The code I used: 

export const s3Test = async () => {
    // debugger;

    const creds = fromCognitoIdentityPool({
        client: new CognitoIdentityClient({ region }),
        identityPoolId: idPool,
    });

    const client = new S3Client({
        region,
        credentials: credentials
    });

    const Key = `${Date.now()}-new-key`;
    let upload = new Upload({
        client,
        tags: [{
            Key: "my-tag-key",
            Value: "my-tag-value"
        }],
        params: { Key, Bucket,
            Body: "hello world!"
        }
    }); 

    upload.on("httpUploadProgress", (progress ) => {
        console.log(progress);
    });

    const uploadResult = await upload.done();

    // debugger;
    console.log("done uploadingg", uploadResult);

}
ajredniwja commented 3 years ago

For Angular using web-pack 5 it is using the node config to build which causes this error, which uses fs as seen here, but it needs to use the browser config as which doesn't use fs. I believe, it should be a web-pack issue since it is not able to load the correct config. Not sure how to achieve that in an Angular project but working on it, will discuss it with the team to find more on the issue.

WIStudent commented 3 years ago

A webpack team member explained the cause of this issue here: https://github.com/webpack/webpack/issues/13498#issuecomment-853815100

Basicly, the browser field inside the package.json is ignored because the module field takes precedence. Instead they recommend defining package exports with different taget environments.

robec1982 commented 2 years ago

ahh scratch that. The build succeeds, but now it's failing at run time 😕

@grothem did you solve this. I'm experiencing the same issue. Webpack is creating the bundle with 3 WARNINGS (no errors) but the resulting bundle doesn't work. It launches an error like this:

webpack://orders-importer-bundle/./node_modules/amazon-sp-api/lib/endpoints.js?:9
    ...readdirSync(__dirname + '/resources').reduce((eps, ep) => {
       ^

TypeError: readdirSync is not a function
grothem commented 2 years ago

@robec1982 It's working for me now with the latest version of the sdk

RanVaknin commented 1 year ago

seems like this is solved based on the last comment and the lack of activity. if this persists please open a new issue.

Thanks, Ran~

github-actions[bot] commented 1 year ago

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs and link to relevant comments in this thread.