SecretsManagerClient looks up credentials using fs.readFile even if credentials are passed when initializing

nwehner commented 3 years ago

Describe the bug

Despite creating a new SecretsManagerClient with credentials included in the initialization options, the program uses fs.readFile to grab credentials off-disk. The SecretsManagerClient should be using the provided credentials and thus not use fs.readFile at all.

Your environment

aws/codebuild/standard:4.0 Linux image with npm@7.7.5 and node@12.18.0

SDK version number

@aws-sdk/client-s3@3.13.0

Is the issue in the browser/Node.js/ReactNative?

Node.js

Details of the browser/Node.js/ReactNative version

Paste output of npx envinfo --browsers or node -v or react-native -v v12.18.0

Steps to reproduce

If you create a new SecretsManagerClient like so new SecretsManagerClient({ region: 'us-west-2', credentials: { accessKeyId: 'myAccessKey', secretAccessKey: 'mySecretKey' } }) and proceed to resolve the Promise, the slurpFile function from @aws-sdk/shared-ini-file-loader is used to grab credentials off the file system.

Observed behavior

While building a new Webpack project with SecretsManagerClient included therein, Webpack warns that there is "Unsafe builtin usage fs.readFile" pointing to the slurfFile function in @aws-sdk/shared-ini-file-loader/dist/es/index.js:87:17

Expected behavior

fs.readFile should not be invoked at all since credentials are passed to the client during initialization.

Additional context

This might possibly be related to https://github.com/aws/aws-sdk-js-v3/issues/2027

AllanZhengYP commented 3 years ago

Hi @nwehner

fs.readFile exists in the SDK no matter whether you provide the credentials. Can you confirm it's acctually called in the credential resolution? because the there are a handful places where fs.readFile will be used. For example, SDK will try to load region from disk if not provided; SDK will also try loading the maxAttempt too if not provided: https://github.com/aws/aws-sdk-js-v3/blob/ca6f146946ff2ff178495fea902c3c18e93420a3/clients/client-secrets-manager/runtimeConfig.ts#L31.

However, they will be read only once at construction.

nwehner commented 3 years ago

Good to know! Is there a list somewhere of all the properties that would need to be populated off-disk?

I made a small edit to the slurpFile function to reject the promise right away, rather than trying to readFile and resolving the promise with that data. When Gatsby's building everything with Webpack, I get an UnhandledPromiseRejectionWarning regarding the rejection I introduced. So, even with credentials and region populated in the SecretsManager options, the program is trying to get something off-disk.

nwehner commented 3 years ago

Did some more logging and I can confirm it's trying to grab my .aws/config file. That said, without the config file I can still retrieve all the secrets just fine, given that the SecretsManagerClient is initialized with region and credentials (accessKeyId & secretAccessKey).

nwehner commented 3 years ago

I tried initializing the SecretsManagerClient with every configuration option I could find, but the program is still trying to grab my .aws/config file...

{ apiVersion: '2017-10-17', region: 'us-west-2', credentials: { accessKeyId: getPublic(), secretAccessKey: getPrivate(), expiration: undefined, sessionToken: undefined }, maxAttempts: 10, tls: true, signer: undefined, signingEscapePath: undefined, systemClockOffset: undefined, signingRegion: undefined, retryStrategy: undefined, endpoint: undefined, customUserAgent: undefined, defaultUserAgentProvider: undefined, base64Decoder: undefined, base64Encoder: undefined, bodyLengthChecker: undefined, credentialDefaultProvider: undefined, streamCollector: undefined, disableHostPrefix: undefined, utf8Decoder: undefined, utf8Encoder: undefined, requestHandler: undefined, regionInfoProvider: undefined, serviceId: undefined, logger: undefined, urlParser: undefined, runtime: undefined, sha256: undefined }

jamiehodge commented 3 years ago

I believe we're seeing the same issue with the S3 client. Setting AWS_SDK_LOAD_CONFIG=0 has no effect.

moltar commented 2 years ago

This causes Lambda to fail on heavy load:

    error: {
      "type": "NodeError",
      "message": "A system error occurred: uv_os_homedir returned EMFILE (too many open files)",
      "stack":
          SystemError [ERR_SYSTEM_ERROR]: A system error occurred: uv_os_homedir returned EMFILE (too many open files)
              at Object.getHomeDir (/node_modules/@aws-sdk/shared-ini-file-loader/dist-cjs/index.js:82:17)
              at Object.loadSharedConfigFiles (/node_modules/@aws-sdk/shared-ini-file-loader/dist-cjs/index.js:11:89)
              at null.<anonymous> (/node_modules/@aws-sdk/node-config-provider/dist-cjs/fromSharedConfigFiles.js:9:53)
              at null.<anonymous> (/node_modules/@aws-sdk/property-provider/dist-cjs/chain.js:11:28)
              at runMicrotasks (<anonymous>)
              at processTicksAndRejections (internal/process/task_queues.js:95:5)
              at null.coalesceProvider (/node_modules/@aws-sdk/property-provider/dist-cjs/memoize.js:13:24)
              at null.isConstant (/node_modules/@aws-sdk/property-provider/dist-cjs/memoize.js:24:28)
      "code": "ERR_SYSTEM_ERROR",
      "info": {
        "errno": -24,
        "code": "EMFILE",
        "message": "too many open files",
        "syscall": "uv_os_homedir"
      },
      "errno": -24,
      "syscall": "uv_os_homedir"
    }

aws / aws-sdk-js-v3