Deno failing with AWS SDK credentials with npm specifier

[soundstep]

This is about using the AWS SDK V3 with Deno and the npm specifier:

import { ListObjectsCommand, S3Client } from 'npm:@aws-sdk/client-s3@3.264.0';
import { fromContainerMetadata } from 'npm:@aws-sdk/credential-providers@3.264.0';

The issue has reported there: https://github.com/aws/aws-sdk-js-v3/issues/4405. The import is working properly, but the usage does not. Can provide help to debug this issue? Thank you in advance.

[bartlomieju]

Duplicate of https://github.com/denoland/deno/issues/16810, let's continue there.

[soundstep]

@bartlomieju I don't think it is the same issue, this is related to credentials. I believe the behavior I reported is related to the following package that is not working with Deno as it is using node-bindings: https://github.com/awslabs/aws-crt-nodejs Can we please reopen?

[soundstep]

Note that the issue with awslabs/aws-crt-nodejs is also why this aws-sdk deno port has been abandoned: https://github.com/christophgysin/aws-sdk-js-v3/issues/38#issuecomment-1400969072

[bartlomieju]

@soundstep any chance you could try with latest Deno (v1.34.1) and let me know if the problem persists? We polyfilled a lot of missing APIs since you opened this issue and I expect it should just work these days.

[soundstep]

Yes, I'll try to check that when I get a chance!

[soundstep]

I can confirm that this works with deno 1.34.1 and EC2 container metadata credentials:

#!/usr/bin/env -S deno run -A --reload

import { ListObjectsCommand, S3Client } from 'npm:@aws-sdk/client-s3@3.264.0';

console.log('No AWS credentials setup');

const client = new S3Client({
    region: Deno.env.get('AWS_REGION') || 'eu-west-1',
});

console.log('S3 client created, executing ListObjectsCommand');

try {
    const res = await client.send(
        new ListObjectsCommand({
            Bucket: '<BUCKET_NAME>',
        }),
    );

    console.log(res);

    Deno.exit();
} catch (err) {
    console.log(err);
    Deno.exit(1);
}

And this was already working with web identity credentials:

import { ListObjectsCommand, S3Client } from 'npm:@aws-sdk/client-s3@3.264.0';

const client = new S3Client({
    region: Deno.env.get('AWS_REGION') || 'eu-west-1',
});

console.log('S3 client created, executing ListObjectsCommand');

try {
    const res = await client.send(
        new ListObjectsCommand({
            Bucket: '<BUCKET_NAME>',
        }),
    );

    console.log('response length:', res.Contents.length);

    Deno.exit();
} catch (err) {
    console.log(err);
    Deno.exit(1);
}

[andykais]

cross-posting another related issue that I came across with the aws sdk and R2 based on their docs for node usage https://developers.cloudflare.com/r2/reference/data-location:

import * as s3 from 'npm:@aws-sdk/client-s3'

const s3_client = new s3.S3({
  region: 'auto',
  endpoint: R2_BASE_URL,
  credentials: {
    accessKeyId: ACCESS_KEY_ID,
    secretAccessKey: ACCESS_KEY_SECRET,
  }
})

const result = await s3_client.createMultipartUpload({
  Bucket: 'MYBUCKET',
  Key: `${Date.now()}`,
})

gives this error:

error: Uncaught (in promise) 400: UnknownError
    at throwDefaultError (file:///home/andrew/node_modules/.deno/@smithy+smithy-client@2.1.16/node_modules/@smithy/smithy-client/dist-cjs/default-error-handler.js:8:22)
    at file:///home/andrew/node_modules/.deno/@smithy+smithy-client@2.1.16/node_modules/@smithy/smithy-client/dist-cjs/default-error-handler.js:18:39
    at de_CreateMultipartUploadCommandError (file:///home/andrew/node_modules/.deno/@aws-sdk+client-s3@3.465.0/node_modules/@aws-sdk/client-s3/dist-cjs/protocols/Aws_restXml.js:3276:12)
    at Object.runMicrotasks (ext:core/01_core.js:790:30)
    at processTicksAndRejections (ext:deno_node/_next_tick.ts:53:10)
    at runNextTicks (ext:deno_node/_next_tick.ts:71:3)
    at eventLoopTick (ext:core/01_core.js:184:21)
    at async file:///home/andrew/node_modules/.deno/@smithy+middleware-serde@2.0.14/node_modules/@smithy/middleware-serde/dist-cjs/deserializerMiddleware.js:7:24
    at async file:///home/andrew/node_modules/.deno/@aws-sdk+middleware-signing@3.465.0/node_modules/@aws-sdk/middleware-signing/dist-cjs/awsAuthMiddleware.js:30:20
    at async file:///home/andrew/node_modules/.deno/@smithy+middleware-retry@2.0.21/node_modules/@smithy/middleware-retry/dist-cjs/retryMiddleware.js:27:46

running the same snippet in node (with a few changes like removing the npm: specifier and wrapping the top-level await) will succeed so I am confident this is a deno specific issue. I have also been able to run S3::listObjects in deno, so it seems to be some specific logic that is hit by this method.

[tazmaniax]

I think this might be related. The following script importing the S3 package from esh.sh and running on Deno 1.39.4 works...

import { S3 } from "https://esm.sh/@aws-sdk/client-s3@3.490.0"

Deno.serve(async (req) => {
  const { name } = await req.json()

  const config = {
    region: Deno.env.get('AWS_REGION') ?? '',
    credentials: {
      accessKeyId: Deno.env.get('AWS_ACCESS_KEY_ID') ?? '',
      secretAccessKey: Deno.env.get('AWS_SECRET_ACCESS_KEY') ?? '',
      sessionToken: Deno.env.get('AWS_SESSION_TOKEN') ?? '',
    }
  }
  console.log('start instantiating S3 client', config)
  const s3 = new S3(config);
  console.log('finish instantiating S3 client')

  const data = {
    message: `Hello ${name}!`,
  }

  return new Response(
    JSON.stringify(data),
    { headers: { "Content-Type": "application/json" } },
  )
})

However, if S3 is imported using the npm specifier...

// import { S3 } from "https://esm.sh/@aws-sdk/client-s3@3.490.0"
import { S3 } from "npm:@aws-sdk/client-s3@3.490.0"

...then the following error occurs

[Info] finish instantiating S3 client

event loop error: TypeError [ERR_INVALID_ARG_TYPE]" argument must be of type string. Received null
    at assertPath (ext:deno_node/path/_util.ts:10:11)
    at join (ext:deno_node/path/_posix.ts:77:5)
    at getCredentialsFilepath (file:///tmp/sb-compile-edge-runtime/node_modules/localhost/@smithy/shared-ini-file-loader/2.2.8/dist-cjs/getCredentialsFilepath.js:7:99)
    at loadSharedConfigFiles (file:///tmp/sb-compile-edge-runtime/node_modules/localhost/@smithy/shared-ini-file-loader/2.2.8/dist-cjs/loadSharedConfigFiles.js:12:76)
    at file:///tmp/sb-compile-edge-runtime/node_modules/localhost/@smithy/node-config-provider/2.1.9/dist-cjs/fromSharedConfigFiles.js:8:102
    at file:///tmp/sb-compile-edge-runtime/node_modules/localhost/@smithy/property-provider/2.0.17/dist-cjs/chain.js:12:39
    at eventLoopTick (ext:core/01_core.js:183:11)
    at async coalesceProvider (file:///tmp/sb-compile-edge-runtime/node_modules/localhost/@smithy/property-provider/2.0.17/dist-cjs/memoize.js:14:24)
    at async file:///tmp/sb-compile-edge-runtime/node_modules/localhost/@smithy/property-provider/2.0.17/dist-cjs/memoize.js:26:28
failed to send request to user worker: connection closed before message completed
InvalidWorkerResponse: user worker failed to respond
    at async Promise.all (index 1)
    at async UserWorker.fetch (ext:sb_user_workers/user_workers.js:64:19)
    at async Server.<anonymous> (file:///home/deno/main/index.ts:146:12)
    at async #respond (https://deno.land/std@0.182.0/http/server.ts:220:18) {
  name: "InvalidWorkerResponse"
}

The two import approaches are loading in different dependencies despite the same version being specified. Interestingly, the same code runs successfully outside of a Deno Deploy function running locally.

[andykais]

fwiw, I moved over to a deno library for interacting with aws (well, R2 compatible apis in my case) https://deno.land/x/aws_api@v0.8.1. This repo isnt quite as maintained as the official npm sdk, but it works like a charm