search

aws / aws-sdk-js-v3

Modularized AWS SDK for JavaScript.
Apache License 2.0
3.12k stars 578 forks source link

AWS typescript SDK for deno typescript runtime. #1289

Closed PaulThompson closed 8 months ago

PaulThompson commented 4 years ago

Is your feature request related to a problem? Please describe. Typescript code generated to a file tree on github, in a manner that can be imported and used by deno. So that users of deno can use the AWS typescript SDK in future.

Describe the solution you'd like Typescript code generated to a file tree on github, in a manner that can be imported by deno. A simple copy of the generated typescript transformed with a find & replace to make imports use a ".ts" suffix on files eg: https://github.com/aws/aws-sdk-js-v3/blob/master/clients/client-s3/S3.ts#L1 would read "import { S3Client } from "./S3Client.ts"; would be a start. Deno aims to be very browser compatible including http fetch API.

Describe alternatives you've considered An alternative process could be to create a separate repo that watches this repo and transforms the typescript sources as required. Another alternative is to address this request upstream to Smithy for typescript code-gen https://github.com/awslabs/smithy-typescript.

Additional context https://deno.land/

PaulThompson commented 4 years ago

Attempted the following:

According to https://www.reddit.com/r/Deno/comments/hcdtrx/deno_faq_for_this_subreddit/ jspm.dev operates a rewriting service that transforms NPM modules.

Therefore tried this deno typescript source:

import S3 from 'https://jspm.dev/@aws-sdk/client-s3';

async function test() {
    const s3 = new S3({region:'ap-southeast-2'});
    const buckets = await s3.listBuckets({});
    console.log(buckets);
}

test();

Unfortunately jspm doesn't implement node's http2 library:

error: Import 'https://jspm.dev/npm:@jspm/core@2/nodelibs/http2' failed: 404 Not Found
Imported from "https://jspm.dev/npm:@aws-sdk/node-http-handler@1!cjs:8"

https://github.com/jspm/jspm-core/issues/5

PaulThompson commented 4 years ago

Apparantly there is a tool https://www.npmjs.com/package/denoify to transform .ts sources.

PaulThompson commented 4 years ago

Working on this in https://github.com/PaulThompson/aws-sdk-js-v3/tree/master/deno

PaulThompson commented 4 years ago

Denoify was too generic for the task - particularly the fact that all the @aws-sdk/* dependencies are right here in the repo. Had an attempt at doing the various find/replaces to get this to work.

See https://github.com/PaulThompson/aws-sdk-js-v3/tree/master/deno

Latest error on that is:

error: TS1192 [ERROR]: Module '"https://raw.githubusercontent.com/PaulThompson/aws-sdk-js-v3/master/deno/client-s3/mod"' has no default export.
import S3 from 'https://raw.githubusercontent.com/PaulThompson/aws-sdk-js-v3/master/deno/client-s3/mod.ts';
       ~~
    at file:///tmp/test_deno_s3.ts:1:8

TS2552 [ERROR]: Cannot find name 'FileReader'. Did you mean 'fileReader'?
    const fileReader = new FileReader();
                           ~~~~~~~~~~
    at https://raw.githubusercontent.com/PaulThompson/aws-sdk-js-v3/master/deno/chunked-blob-reader/mod.ts:7:28

    'fileReader' is declared here.
        const fileReader = new FileReader();
              ~~~~~~~~~~
        at https://raw.githubusercontent.com/PaulThompson/aws-sdk-js-v3/master/deno/chunked-blob-reader/mod.ts:7:11

Which is unfortunately a deno missing API. https://github.com/denoland/deno/issues/5249

nl-brett-stime commented 4 years ago

The fetch-http-handler should work with Deno:

https://github.com/aws/aws-sdk-js-v3/tree/81b2e87067642a8cea8649cbdb2c342ca9fb6ac6/packages/fetch-http-handler

I tried loading the EventBridge client via the pika.dev service to no avail: https://www.pika.dev/npm/@aws-sdk/client-eventbridge

nl-brett-stime commented 4 years ago

Deno support would be awesome for scripts that are easy to pass around without having to pass a whole project directory and without requiring colleagues to have deep knowledge of e.g., pip (Python), bundler (Ruby), npm/yarn, etc.

nl-brett-stime commented 4 years ago

It's not pretty but, for anyone who comes looking, I was able to piece together a request that seems to mostly* work:

// To Run: deno run --allow-net /path/to/this/file.ts

import FetchHttpHandler from 'https://jspm.dev/@aws-sdk/fetch-http-handler';
import AwsTypes from 'https://jspm.dev/@aws-sdk/types'
import SignatureV4 from 'https://jspm.dev/@aws-sdk/signature-v4';
import Sha256 from "https://jspm.dev/@aws-crypto/sha256-js";
import moment from 'https://cdn.skypack.dev/moment';

const signer = new SignatureV4.SignatureV4({
    credentials: {
        accessKeyId: 'SomeAccessKey',
        secretAccessKey: 'SomeSecretKey',
        sessionToken: 'SomeSessionToken'
    },
    region: "eu-west-2",
    service: "events",

    sha256: Sha256.Sha256
});

var request = {
    method: "POST",
    headers: {
        "X-Amz-Target": "AWSEvents.PutEvents",
        "Content-Type": "application/x-amz-json-1.1"
    },
    body: JSON.stringify({
        "Entries": [
            {
                "DetailType": "Scheduled Event",
                "Source": "aws.events",
                "Resources": [ "arn:aws:events:eu-west-2:SomeAccountId:rule/SomeRuleName" ],
                "Account": "SomeAccountId",
                "Time": moment().valueOf(),
                "Region": "eu-west-2",
                "Detail": "{}"
            }
        ]
    }),

    protocol: "https:",
    hostname: "events.eu-west-2.amazonaws.com",
    port: 443,
    path: "/",
    query: null,
}

request = await signer.sign(request);

const fetcher = new FetchHttpHandler.FetchHttpHandler();
const response = await fetcher.handle(request);
const parsedResponse = await response.body.text();

console.log('response', parsedResponse);

* The Deno code above does seem to reach the right backend logic at AWS but the particular action I was trying to invoke is apparently not allowed by AWS. When I ran the code with expired credentials, I got the following response: {"__type":"ExpiredTokenException","message":"The security token included in the request is expired"}. After I fixed that and another minor issue about the format of the timestamp in the body of my request, I got the following response: {"Entries":[{"ErrorCode":"NotAuthorizedForSourceException","ErrorMessage":"Not authorized for the source."}],"FailedEntryCount":1} which means that my use-case of manually testing a Cron/Schedule rule won't work (even using a more mature client) because my user account, which did seem to authenticate, is not authorized to impersonate the EventBridge service itself.

christophgysin commented 4 years ago

deno FileReader API has been merged to master: https://github.com/denoland/deno/issues/5249 :champagne:

I forked the work from @PaulThompson and added a few more fixes. We can now make some basic API requests:

import { S3 } from 'https://raw.githubusercontent.com/christophgysin/aws-sdk-js-v3/deno/deno/client-s3/mod.ts'

const s3 = new S3({
  region: Deno.env.get('AWS_REGION'),
  credentials: {
    accessKeyId: Deno.env.get('AWS_ACCESS_KEY_ID')!,
    secretAccessKey: Deno.env.get('AWS_SECRET_ACCESS_KEY')!,
  },
})

const { Buckets = [] } = await s3.listBuckets({})
Buckets.forEach(bucket => console.log(bucket.Name))
PaulThompson commented 4 years ago

Thanks @christophgysin Yes and now deno 1.3.0 has the FileReader landed.

hom-bahrani commented 4 years ago

someone has been working on DynamoDB client https://github.com/chiefbiiko/dynamodb

mrowles commented 3 years ago

I noticed SecretsManager > listSecrets request params are missing 'Filter' param.

const listSecretsRequest: ListSecretsRequest = {
    // this errors out
    Filters: [{
        Key: 'name',
        Values: ['someSecretName],
    }],
};

const existingSecretsResponse: ListSecretsResponse = await secretsManager.listSecrets(
    listSecretsRequest,
);

Technically, listSecrets() function wants ListSecretsCommandInput, but this is just a type that is set to ListSecretsRequest.

I'd love to help, but not sure where to start!

christophgysin commented 3 years ago

@mrowles Could you open an issue on https://github.com/christophgysin/aws-sdk-js-v3/issues? Please provide a full example to reproduce the problem.

mrowles commented 3 years ago

@christophgysin Done, cheers - https://github.com/christophgysin/aws-sdk-js-v3/issues/3

ghost commented 3 years ago

Straight question as the outcome of this thread is not clear ..... does the AWS JavaScript SDK fully support Deno?

christophgysin commented 3 years ago

@bootrino No, it does not support deno at all. I maintain a fork that I try to keep in sync which supports deno, and is published on deno.land.

ghost commented 3 years ago

That's a pity, anyone know if Amazon plans to officially support deno? Is posting in this thread making such a request, or does such a request need to be posted elsewhere?

trivikr commented 3 years ago

Is posting in this thread making such a request, or does such a request need to be posted elsewhere?

This is maintainer of AWS SDK for JavaScript.

Tagging recent commenters @bootrino @christophgysin @mrowles Do upvote on the first post on this issue https://github.com/aws/aws-sdk-js-v3/issues/1289#issue-644050906, so that it appears in one of the most voted issues.

Some ways this request can get prioritized is:

PaulThompson commented 3 years ago

Thanks for the encouragement @trivikr and the work @christophgysin on https://github.com/christophgysin/aws-sdk-js-v3 https://deno.land/x/aws_sdk

trivikr commented 3 years ago

Update: The private Deno company was just announced https://deno.com/blog/the-deno-company

Rest assured that Deno will remain MIT licensed. For Deno to grow and be maximally useful, it must remain permissively free. We don’t believe the “open core” business model is right for a programming platform like Deno. We do not want to find ourselves in the unfortunate position where we have to decide if certain features are for paid customers only. If you watch our conference talks, you will find we've been hinting at commercial applications of this infrastructure for years. We are bullish about the technology stack we've built and intend to pursue those commercial applications ourselves. Our business will build on the open source project, not attempt to monetize it directly.

lewisl9029 commented 2 years ago

@trivikr As of time of writing, this issue has the highest number of 👍s by a fairly large margin: https://github.com/aws/aws-sdk-js-v3/issues?q=is%3Aissue+is%3Aopen+sort%3Areactions-%2B1-desc

Great work deno community! 🎉

I can't speak to how much progress has been made on the other points, but for my use case, Deno's number 1 advantage compared to node is its excellent support for HTTP2 out of the box with its native fetch for the client side and built-in http server APIs for the server side.

From my limited research so far, the HTTP2 client situation in node is especially dire:

Many of the most widely used AWS services (dynamo, s3, lambda, etc) would benefit greatly from a wider availability of high-quality HTTP2-capable clients (fewer tcp connections -> lower latency for clients and lower load on servers). In fact, this library itself can significantly reduce its maintenance burden in the long term by relying on a robust native fetch implementation in deno over maintaining its own http2 client built on top of the low-level APIs provided by node (and hope node eventually catches up with a native fetch implementation of its own).

trivikr commented 2 years ago

Deno v1.25 now has experimental support for npm.

If you test AWS SDK for JavaScript (v3) with Deno v1.25, do post your summary in this issue. If any Node.js modules are not supported on Deno, report those problems on Deno issue tracker.

vicary commented 2 years ago

Deno v1.25 now has experimental support for npm.

It's just not ready yet, their node polyfill in std didn't implement Hmac. Despite there is a Hmac module available 3rd party, the npm: resolution mechanism has no way to override that via import map.

Given the following script:

import { S3 } from "npm:@aws-sdk/client-s3";

await new S3().listBuckets({});

Deno will fetch and cache all the node dependencies, and then throw this error in runtime:

error: Uncaught Error: Not implemented: crypto.Hmac
  throw new Error(message);
        ^
    at notImplemented (https://deno.land/std@0.155.0/node/_utils.ts:23:9)
    at new Hmac (https://deno.land/std@0.155.0/node/internal/crypto/hash.ts:136:5)
    at createHmac (https://deno.land/std@0.155.0/node/crypto.ts:264:10)
    at new Hash (file:///Users/vicary/Library/Caches/deno/npm/registry.npmjs.org/@aws-sdk/hash-node/3.171.0/dist-cjs/index.js:9:54)
    at hmac (file:///Users/vicary/Library/Caches/deno/npm/registry.npmjs.org/@aws-sdk/signature-v4/3.171.0/dist-cjs/credentialDerivation.js:35:18)
    at getSigningKey (file:///Users/vicary/Library/Caches/deno/npm/registry.npmjs.org/@aws-sdk/signature-v4/3.171.0/dist-cjs/credentialDerivation.js:11:29)
    at SignatureV4.getSigningKey (file:///Users/vicary/Library/Caches/deno/npm/registry.npmjs.org/@aws-sdk/signature-v4/3.171.0/dist-cjs/SignatureV4.js:156:57)
    at SignatureV4.signRequest (file:///Users/vicary/Library/Caches/deno/npm/registry.npmjs.org/@aws-sdk/signature-v4/3.171.0/dist-cjs/SignatureV4.js:101:73)
    at async file:///Users/vicary/Library/Caches/deno/npm/registry.npmjs.org/@aws-sdk/middleware-signing/3.171.0/dist-cjs/middleware.js:13:18
    at async StandardRetryStrategy.retry (file:///Users/vicary/Library/Caches/deno/npm/registry.npmjs.org/@aws-sdk/middleware-retry/3.171.0/dist-cjs/StandardRetryStrategy.js:51:46)
mlafeldt commented 2 years ago

After publishing this post, someone from Deno pointed me to aws-api.deno.dev, which turned out to provide the only DynamoDB client for Deno that worked for me right off the bat. Maybe worth considering as an alternative until we can use the official SDK.

danopia commented 2 years ago

someone from Deno pointed me to aws-api.deno.dev, which turned out to provide the only DynamoDB client for Deno that worked for me right off the bat. Maybe worth considering as an alternative until we can use the official SDK.

Thanks for the shoutout! My aws-api.deno.dev project is intended to serve as a minimal Deno-first API client for all of the various AWS services. It certainly lacks various higher-level routines, notably DynamoDB document helpers (as shown in the linked tweet) and S3 pre-signing.

Anyway, aws-api.deno.dev should work for most API-invoking usecases, and I still use it extensively, but I hope to see it made irrelevant by first-class SDK support from AWS ❤️

skorfmann commented 1 year ago

in case it helps someone: got the following to work in the current Deno version:

import { PutObjectCommand, S3Client } from "https://esm.sh/@aws-sdk/client-s3@3.245.0"
// trying to load the credentials provider through esm.sh as well didn't work
import { fromSSO } from 'npm:@aws-sdk/credential-providers'
const s3 = new S3Client({ credentials, region: 'eu-central-1'})

await s3.send(new PutObjectCommand({
      Bucket: "myBucket",
      Key: '/my/key',
      Body: 'Body'
}))

The same works for the Lambda client.

soundstep commented 1 year ago

@skorfmann This did not work with aws-sdk 3.264.0, with either esm.sh or the deno npm specifier, using fromContainerMetadata or fromInstanceMetadata.

cstayyab commented 1 year ago

Using npm specifier in deno, it is unable to import an internal package hash-node. Here's the error:

error: Uncaught TypeError: Expected a JavaScript or TypeScript module, but identified a Unknown module. Importing these types of modules is currently not supported.
  Specifier: https://jspm.dev/npm:@aws-sdk/hash-node@3.267.0
    at https://jspm.dev/@aws-sdk/hash-node:7:15
        const exports = await import(path.join(file.path, file.name))
harshalone commented 1 year ago

hi @christophgysin I tried to import_map

"aws-ses-client" : "https://deno.land/x/aws_sdk@v3.6.0.0/client-ses?source",

and then

in my edge function

import { SendEmailCommand, SESClient } from "aws-ses-client";

but I am getting this error:

Error: Module not found "https://deno.land/x/aws_sdk@v3.6.0.0/client-ses?source".

harshalone commented 1 year ago

in case it helps someone: got the following to work in the current Deno version:

import { PutObjectCommand, S3Client } from "https://esm.sh/@aws-sdk/client-s3@3.245.0"
// trying to load the credentials provider through esm.sh as well didn't work
import { fromSSO } from 'npm:@aws-sdk/credential-providers'
const s3 = new S3Client({ credentials, region: 'eu-central-1'})

await s3.send(new PutObjectCommand({
      Bucket: "myBucket",
      Key: '/my/key',
      Body: 'Body'
}))

The same works for the Lambda client.

hi can I ask you how did you find this esm .sh link?

I want to know if there is a search engine where I can search for different esm modules somewhere?

Also thanks allot for sharing it :)

soundstep commented 1 year ago

For the version you mean? If yes, just visit https://esm.sh/@aws-sdk/client-s3 will bring you to the latest version https://esm.sh/@aws-sdk/client-s3@3.400.0. If you are looking for other ESM bundlers, here are a few (but they are not all equal): https://esm.sh/ https://www.jsdelivr.com/ https://www.skypack.dev/ https://jspm.org/ https://cdnjs.com/ https://unpkg.com/ https://esb.deno.dev/ https://denopkg.com/

pradeepb28 commented 10 months ago

hi @christophgysin I tried to import_map

"aws-ses-client" : "https://deno.land/x/aws_sdk@v3.6.0.0/client-ses?source",

and then

in my edge function

import { SendEmailCommand, SESClient } from "aws-ses-client";

but I am getting this error:

Error: Module not found "https://deno.land/x/aws_sdk@v3.6.0.0/client-ses?source".

Did you find a solution?

vicary commented 10 months ago

Updates on my last comment. As of Deno 1.39.0, npm support already allows the use of AWS SDK.

The following snippet already works.

import { S3 } from "npm:@aws-sdk/client-s3";

await new S3({}).listBuckets({});
RanVaknin commented 8 months ago

Hi everyone on the thread. This feature request was never meant for the AWS SDK to address, the AWS SDK for JavaScript only officially supports the Node runtime, therefore it was up to Deno, to make their runtime compatible with Node's.

Based on @vicary 's last comment I assume this is now the case, but I haven't actually looked into it.

Because of those factors, I'm going to go ahead and close this issue. Since there is a lot of comments and upvotes on this thread, if anything is left unresolved, please feel free to create a separate issue so we can address it on a per-customer cadence.

Thanks again, Ran~

PaulThompson commented 8 months ago

Understood, @RanVaknin.

To note though, the AWS SDK does actually have separate support for node, browser and react-native (see all runtimeConfig*.ts files). At the time of the original post was looking like deno would be a variant needing minor differences that could be accommodated in a similar manner.

Over time deno has added support for importing npm modules as well as supporting more of node's own internal APIs to the point where it now works.

test.ts:

import { S3 } from "npm:@aws-sdk/client-s3";
import { SecretsManager } from "npm:@aws-sdk/client-secrets-manager";

console.log(
await new S3({
  region:"ap-southeast-2"
}).listBuckets({}));

console.log(
await new SecretsManager({
  region:"ap-southeast-2"
}).listSecrets({}));
deno run --allow-sys --allow-env --allow-net --allow-read test.ts
github-actions[bot] commented 8 months ago

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs and link to relevant comments in this thread.