Instance and container metadata credentials not working with Deno

soundstep commented 1 year ago

Checkboxes for prior research

[X] I've gone through Developer Guide and API reference
[X] I've checked AWS Forums and StackOverflow.
[X] I've searched for previous similar issues and didn't find any solution.

Describe the bug

Automatic credentials fromInstanceMetadata and fromContainerMetadata are not picked up and is hanging (no error message, no timeout) when used with Deno. This happens on our CI in 2 different Jenkins instances (one EC2, one EKS), it does work locally with environment variables, or on the CI using NodeJS. Note: I realize the Deno usage is not official, but any help would be greatly appreciated to either resolve the issue or debug it. Deno has been a tremendous improvement in our workflows.

SDK version number

@aws-sdk/client-s3@3.264.0

Which JavaScript Runtime is this issue in?

Deno

Details of the browser/Node.js/ReactNative version

deno 1.30.2 (release, aarch64-apple-darwin)
v8 10.9.194.5
typescript 4.9.4

Reproduction Steps

Note that the following scripts are working correctly with this port, but this port has been abandoned: https://github.com/christophgysin/aws-sdk-js-v3 (https://deno.land/x/aws_sdk@v3.32.0-1/client-s3/mod.ts)

Test no credentials setup

Install Deno
Create a deno script main.ts:

import { ListObjectsCommand, S3Client } from 'npm:@aws-sdk/client-s3@3.264.0';

console.log('No AWS credentials setup');

const client = new S3Client({
    region: 'eu-west-1'
});

console.log('S3 client created, executing ListObjectsCommand');

try {
    const res = await client.send(
        new ListObjectsCommand({
            Bucket: 'my-s3-bucket'
        }),
    );

    console.log(res);

    Deno.exit();
} catch (err) {
    console.log(err);
    Deno.exit(1);
}

Run the script: deno run -A --reload main.ts

Output:

The script is running and hanging forever:

Test with credentials setup

Install Deno
Create a deno script main-container.ts:

import { ListObjectsCommand, S3Client } from 'npm:@aws-sdk/client-s3@3.264.0';
import { fromContainerMetadata } from 'npm:@aws-sdk/credential-providers@3.264.0';

console.log('Using fromContainerMetadata credentials');

const client = new S3Client({
    region: 'eu-west-1',
    credentials = fromContainerMetadata({ timeout: 2000, maxRetries: 1 })
});

console.log('S3 client created, executing ListObjectsCommand');

try {
    const res = await client.send(
        new ListObjectsCommand({
            Bucket: 'my-s3-bucket'
        }),
    );

    console.log(res);

    Deno.exit();
} catch (err) {
    console.log(err);
    Deno.exit(1);
}

Run the script: deno run -A --reload main-container.ts

Output:

The script is running and hanging forever:

Test with credentials setup and `esm.sh`

Install Deno
Create a deno script main-esmsh.ts:

import { ListObjectsCommand, S3Client } from 'https://esm.sh/@aws-sdk/client-s3@3.264.0';
// Note: fromContainerMetadata is not available from https://esm.sh/@aws-sdk/credential-providers@3.264.0
import { fromContainerMetadata } from 'https://esm.sh/@aws-sdk/credential-provider-imds@3.259.0';

console.log('Using fromContainerMetadata credentials');

const client = new S3Client({
    region: 'eu-west-1',
    credentials = fromContainerMetadata({ timeout: 2000, maxRetries: 1 })
});

console.log('S3 client created, executing ListObjectsCommand');

try {
    const res = await client.send(
        new ListObjectsCommand({
            Bucket: 'my-s3-bucket'
        }),
    );

    console.log(res);

    Deno.exit();
} catch (err) {
    console.log(err);
    Deno.exit(1);
}

Run the script: deno run -A --reload main-esmsh.ts

Output:

The script output an error:

11:20:43  Using fromContainerMetadata credentials
11:20:43  S3 client created, executing ListObjectsCommand
11:20:43  Error: Credential is missing
11:20:43      at re.credentialProvider (https://esm.sh/v106/@aws-sdk/client-s3@3.264.0/deno/client-s3.js:4:183903)
11:20:43      at re.signRequest (https://esm.sh/v106/@aws-sdk/signature-v4@3.257.0/deno/signature-v4.js:3:522)
11:20:43      at re.sign (https://esm.sh/v106/@aws-sdk/signature-v4@3.257.0/deno/signature-v4.js:2:4709)
11:20:43      at sign (https://esm.sh/v106/@aws-sdk/signature-v4-multi-region@3.264.0/deno/signature-v4-multi-region.js:2:764)
11:20:43      at https://esm.sh/v106/@aws-sdk/middleware-signing@3.257.0/deno/middleware-signing.js:2:2366
11:20:43      at async https://esm.sh/v106/@aws-sdk/middleware-retry@3.259.0/deno/middleware-retry.js:2:5465
11:20:43      at async https://esm.sh/v106/@aws-sdk/middleware-logger@3.257.0/deno/middleware-logger.js:2:34
11:20:43      at async file:///tmp/jenkins/workspace/Playground/scriptTest/test-ec2-credentials/esmsh/main-container-esmsh.ts:26:17

Observed Behavior

With a script executing a simple ListObjectsCommand on an S3 Bucket (scripts are provided in the reproduction step).

The script is hanging forever with either (no error, no output, no timeout):

no credentials set
the npm: specifier usage (deno npm specifier)

Output:

The script outputs an error if we use an esm.sh URL (see reproduction step).

import { ListObjectsCommand, S3Client } from 'https://esm.sh/@aws-sdk/client-s3@3.264.0';
import { fromContainerMetadata } from 'https://esm.sh/@aws-sdk/credential-provider-imds@3.259.0';

Note that I had to use @aws-sdk/credential-provider-imds instead of @aws-sdk/credential-providers, as with esm.sh, the fromContainerMetadata export is not available.

Expected Behavior

To be logged in automatically.

Possible Solution

No response

Additional Information/Context

Deno version:

deno 1.30.2 (release, aarch64-apple-darwin)
v8 10.9.194.5
typescript 4.9.4

OS: Mac OS 12.1, Linux alpine, Linux Centos

christophgysin commented 1 year ago

This was fixed in my fork using a custom implementation for deno: https://github.com/christophgysin/aws-sdk-js-v3/tree/deno/packages/credential-provider-imds-deno

soundstep commented 1 year ago

Oh great, thank you for the info!

soundstep commented 1 year ago

Am I wrong in thinking that the behavior I reported is related to the following package that is not working with Deno as it is using node bindings? https://github.com/awslabs/aws-crt-nodejs This is used in the package @aws-sdk/signature-v4-crt, not quite sure how it is related to the rest. Also mentioned there: https://github.com/christophgysin/aws-sdk-js-v3/issues/38#issuecomment-1400969072 Is there any workaround?

yenfryherrerafeliz commented 1 year ago

Hi @soundstep, thanks for opening this issue. I was able to reproduce the reported behavior, however, I am not very familiar with Deno runtime, and I am currently working on understanding what exactly maybe causing the issue here.

I will get back to you as soon as possible.

Thanks!

cdeck95 commented 1 year ago

@soundstep did you find a solution? I am running into the same problem. If so, do you mind showing your updated code or package references?

yenfryherrerafeliz commented 1 year ago

Hi @soundstep, @cdeck95, sorry for the delay answering on this. Basically, this issue is caused by the http library that deno uses, which seems to be different from the one nodejs provides and therefore the request sent is rejected by the imds. Since we do not support Deno runtime, one thing you folks can do is to do a custom implementation to fetch the credentials from the instance metadata service. Actually that is what @christophgysin did. Here is also the documentation.

What I can do now is to mark this issue as feature request to gather community sentiments about this and take further actions based on it.

Thanks!

soundstep commented 1 year ago

Thank you @yenfryherrerafeliz for looking into this.

I think the reason why @christophgysin port has been abandoned is he cannot compile his sdk port anymore because of an underlying crt library, more info there: https://github.com/christophgysin/aws-sdk-js-v3/issues/38#issuecomment-1400969072

We have other EKS containers that run on web identity token, I will try to provide these credentials manually once I get the info I need to see how it goes.

@cdeck95 No solution so far I'm afraid, FYI see the IMDS port from @christophgysin there.

soundstep commented 1 year ago

FYI, this works with web identity token in containers (EKS), the credentials are picked up correctly:

import { ListObjectsCommand, S3Client } from 'npm:@aws-sdk/client-s3@3.264.0';

const client = new S3Client({
    region: Deno.env.get('AWS_REGION') || 'eu-west-1',
});

const res = await client.send(
    new ListObjectsCommand({
        Bucket: 'my-bucket',
    }),
);

console.log('response length:', res.Contents.length);

I'm not sure I will have the time to look into IMDS (container and instance metadata) for the time being, which is not working.

andykais commented 9 months ago

another error that seems to be deno-specific:

import * as s3 from 'npm:@aws-sdk/client-s3'

const s3_client = new s3.S3({
  region: 'auto',
  endpoint: R2_BASE_URL,
  credentials: {
    accessKeyId: ACCESS_KEY_ID,
    secretAccessKey: ACCESS_KEY_SECRET,
  }
})

const result = await s3_client.createMultipartUpload({
  Bucket: 'MYBUCKET',
  Key: `${Date.now()}`,
})

gives this error:

error: Uncaught (in promise) 400: UnknownError
    at throwDefaultError (file:///home/andrew/node_modules/.deno/@smithy+smithy-client@2.1.16/node_modules/@smithy/smithy-client/dist-cjs/default-error-handler.js:8:22)
    at file:///home/andrew/node_modules/.deno/@smithy+smithy-client@2.1.16/node_modules/@smithy/smithy-client/dist-cjs/default-error-handler.js:18:39
    at de_CreateMultipartUploadCommandError (file:///home/andrew/node_modules/.deno/@aws-sdk+client-s3@3.465.0/node_modules/@aws-sdk/client-s3/dist-cjs/protocols/Aws_restXml.js:3276:12)
    at Object.runMicrotasks (ext:core/01_core.js:790:30)
    at processTicksAndRejections (ext:deno_node/_next_tick.ts:53:10)
    at runNextTicks (ext:deno_node/_next_tick.ts:71:3)
    at eventLoopTick (ext:core/01_core.js:184:21)
    at async file:///home/andrew/node_modules/.deno/@smithy+middleware-serde@2.0.14/node_modules/@smithy/middleware-serde/dist-cjs/deserializerMiddleware.js:7:24
    at async file:///home/andrew/node_modules/.deno/@aws-sdk+middleware-signing@3.465.0/node_modules/@aws-sdk/middleware-signing/dist-cjs/awsAuthMiddleware.js:30:20
    at async file:///home/andrew/node_modules/.deno/@smithy+middleware-retry@2.0.21/node_modules/@smithy/middleware-retry/dist-cjs/retryMiddleware.js:27:46

running the same snippet in node (with a few changes like removing the npm: specifier and wrapping the top-level await) will succeed so I am confident this is a deno specific issue. I have also been able to run S3::listObjects in deno, so it seems to be some specific logic that is hit by this method.

This is using R2 so dont crucify me :sweat_smile:. This is how they would normally call your api from node https://developers.cloudflare.com/r2/reference/data-location/

[edit] one more datapoint: I can still successfully use the old sdk.

// @deno-types="npm:aws-sdk/clients/s3.d.ts"
import aws_sdk_s3 from 'npm:aws-sdk/clients/s3.js';

const s3_client = new aws_sdk_s3({
  accessKeyId: ACCESS_KEY_ID,
  secretAccessKey: ACCESS_KEY_SECRET,
  endpoint: R2_BASE_URL,
})

const result = await new Promise((resolve, reject) => {
  s3_client.createMultipartUpload({
    Bucket: 'MYBUCKET',
    Key: `${Date.now()}`,
  }, (error, result) => {
    if (error) reject(error)
    else resolve(result)
  })
})

aBurmeseDev commented 5 months ago

Hey all - checking in here.

This issue seems to be third-party related which we do not support as previously mentioned by my colleague here: https://github.com/aws/aws-sdk-js-v3/issues/4405#issuecomment-1436145300

Please feel free to refer to the workarounds mentioned in the thread and we'll be closing this as not planned.

github-actions[bot] commented 4 months ago

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs and link to relevant comments in this thread.

aws / aws-sdk-js-v3