search

aws / aws-sdk-js-v3

Modularized AWS SDK for JavaScript.
Apache License 2.0
3.03k stars 568 forks source link

S3 Multi-Region Access Point(MRAP) is not available unless with additional dependency #2822

Open AllanZhengYP opened 2 years ago

AllanZhengYP commented 2 years ago

Description

Package @aws-sdk/client-s3 launched the support for S3 Multi-Region Access Point (MRAP) feature in v3.31.0. This feature relies on signer class from aws-crt package. This dependency requires native addons that takes up to 10.5 MB on disk. Because the dependency is only required by MRAP feature, we are omitting the dependency in @aws-sdk/client-s3.

To use this feature, you need to install this dependency explicitly in your application: 

$ npm install @aws-sdk/signature-v4-crt
$ yarn add @aws-sdk/signature-v4-crt

Because aws-crt contains platform-specific binary code, the following use cases are not supported in MRAP:

The suggested use case for supporting MRAP in Node.js app is to deploy your application with the complete node_modules folder.

Describe alternatives you've considered

We are working on solutions to unblock the MRAP for all the supported platforms.

Related issues:

seriousme commented 2 years ago

FYI: One might consider porting the signer class of the common runtime to webassembly. Web assembly is supported in all major browsers and in NodeJs 8+ See also: https://developer.mozilla.org/en-US/docs/WebAssembly/C_to_wasm

Kind regards, Hans

AllanZhengYP commented 2 years ago

Thank you for the advice! @seriousme

We are currently weigh in the options including WASM, ASM.js, and pure JS with WebCrypto. Currently one of the obstacle is that WASM seems not supported in all React Native platforms, so we might need to fallback to use ASM.js in React Native.

On the other hand, we need to make sure the signer is cryptographically secure. If the implementation involves 3rd party crypto primitives, we need to audit them. For example, the signing algorithm in SigV4a requires constant time implementation.

seriousme commented 2 years ago

Good to see you have options :-)

You can mix and match, e.g. use the browsers webcrypto from WASM ;-) e.g. https://ihsavru.medium.com/calling-javascript-code-from-c-c-using-webassembly-a9445c11bc6d

Kind regards, Hans

christophgysin commented 2 years ago

Adding aws-crt blocks me from updating the deno port of aws-sdk-js-v3 past v3.33.0.

:+1: for WASM, which would also allow using this in deno.

rossng commented 2 years ago

Just want to weigh in with the impact this issue currently has on our team. Hopefully it helps to motivate a fix. Our developers use a mix of macOS and Windows machines.

We depend on the @aws-sdk/client-s3 package. This causes the package manager to complain about the missing @aws-sdk/signature-v4-crt peer dependency.

The Windows users go ahead and add the dep to package.json, and everything appears to be resolved. Then a macOS user pulls the latest changes and suddenly package installation is completely broken.

This isn't a great experience. Given that @aws-sdk/signature-v4-crt has ~63k downloads per week, I can only imagine how many developers have run headlong into this so far.

jekh commented 2 years ago

I just ran into this issue, seeing the same peer dependency warnings as @rossng. Using the peerDependenciesMeta field of package.json might be a good solution: https://docs.npmjs.com/cli/v8/configuring-npm/package-json#peerdependenciesmeta

Adding this to package.json in middleware-sdk-s3 should fix this:

"peerDependenciesMeta": {
   "@aws-sdk/signature-v4-crt": {
     "optional": true
   }
 }
AllanZhengYP commented 2 years ago

@jekh Thanks a lot for pointing this out! It's a miss on our side. I will address them a separate PR.

CraigStuntzWillowTree commented 2 years ago

I just tried this with 3.58.0 and I'm still seeing the error on macOS:

WARNING in ../node_modules/@aws-sdk/util-user-agent-node/dist-es/is-crt-available.js 3:96-114
Module not found: Error: Can't resolve 'aws-crt' in '/Users/craig/Documents/PMI/node_modules/@aws-sdk/util-user-agent-node/dist-es'
 @ ../node_modules/@aws-sdk/util-user-agent-node/dist-es/index.js 5:0-52 16:23-37
 @ ../node_modules/@aws-sdk/client-ssm/dist-es/runtimeConfig.js 12:0-65 22:898-914
 @ ../node_modules/@aws-sdk/client-ssm/dist-es/SSMClient.js 10:0-73 15:24-42
 @ ../node_modules/@aws-sdk/client-ssm/dist-es/index.js 2:0-28 2:0-28
...

...where I have the library referenced like so:

    "@aws-sdk/client-ssm": "^3.58.0"

and it's warning on this line:

import { SSM } from "@aws-sdk/client-ssm";
Hasan-Jawaheri commented 2 years ago

This issue is affecting our team and our entire organization will run into the same issue soon as we are moving towards a DNS regional failover which requires SigV4A signing in the browser to make API calls to unkown AWS region (using @aws-sdk/signature-v4-crt).

Are there alternatives for the time being? Can we do SigV4A signing with existing parts of the SDK that work on browser?

samthompsonkennedy commented 2 years ago

This is insane. No update from Amazon? This has put an entire build on hold.

Minimal documentation for MRAP as it is.

ashley-jisc commented 1 year ago

Still nothing for this?

JClackett commented 1 year ago

any updates on this?

anta-semenov commented 1 year ago

any updates on this?

We're trying to use apigateway in multiple regions with custom domains with iam authorization. The request can be routed to different region based on latency, so regular sigV4 doesn't work.

Is there any other workaround to make apigateway works in multiple regions with the same domain name?

anta-semenov commented 1 year ago

@AllanZhengYP I'm curious does c++ code has any platform specific dependencies? It seems like the latest version of react native has first class support c++ code with turbomodules. Does the team consider the option to create a separate rn library for that

summaarum commented 1 year ago

Would also like like updates on this. Getting error during build Module not found: Can't resolve '@aws-sdk/signature-v4-crt' on latest version 3.354.0

MRdNk commented 1 year ago

I'm getting the following error in CloudWatch, when trying to do a putObjectCommand to an MRAP...

ERROR   Invoke Error    
{
    "errorType": "TypeError",
    "errorMessage": "Cannot read properties of undefined (reading 'enable_logging')",
    "$metadata": {
        "attempts": 1,
        "totalRetryDelay": 0
    },
    "stack": [
        "TypeError: Cannot read properties of undefined (reading 'enable_logging')",
        "    at new CrtSignerV4 (/var/task/[bundleName].js:41108:22)",
        "    at SignatureV4MultiRegion.getSigv4aSigner (/var/task/[bundleName].js:41056:33)",
        "    at SignatureV4MultiRegion.sign (/var/task/announcementHandlerBundle.js:41029:25)",
        "    at /var/task/[bundleName].js:9155:31",
        "    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)",
        "    at async /var/task/[bundleName].js:6896:46",
        "    at async /var/task/[bundleName].js:54832:20",
        "    at async /var/task/[bundleName].js:5636:26"
    ]
}

Using the latest S3 and cert lib., in my package.json ...

"@aws-sdk/client-s3": "^3.391.0",
"@aws-sdk/signature-v4-crt": "^3.391.0",

S3Client setup as follows (in side a Ts Lambda function, which works if I just use a standard s3 bucket):

const s3Client = new S3Client({ useGlobalEndpoint: true });

I've tried with and without useGlobalEndpoint; as well as with and without setting the region: 'eu-west-2'; the region the lambda is running in - all the same issue.

NB. Bucket name is set as the MRAP: arn:aws:s3::123456789:accesspoint/[alias].mrap

Any help appreciated.

MRdNk commented 1 year ago

Just read the doesn’t work with bundled nodejs applications; with advice to copy in the node_modules. That’s not going to be easy

MicrosoftJones commented 1 year ago

Patch

kuhe commented 11 months ago

The AWS SDK for JavaScript team is planning to change the usage of the optional CRT dependency in a future update.

The change is described in this announcement: https://github.com/aws/aws-sdk-js-v3/issues/5229.

The change is aimed at reducing errors encountered by bundlers and other static analysis tools. The @aws-sdk/signature-v4-crt package remains optional and is only required when using S3 MRAP or EventBridge.

Legacy131313 commented 11 months ago

Description

Package @aws-sdk/client-s3 launched the support for S3 Multi-Region Access Point (MRAP) feature in v3.31.0. This feature relies on signer class from aws-crt package. This dependency requires native addons that takes up to 10.5 MB on disk. Because the dependency is only required by MRAP feature, we are omitting the dependency in @aws-sdk/client-s3.

To use this feature, you need to install this dependency explicitly in your application:

$ npm install @aws-sdk/signature-v4-crt
$ yarn add @aws-sdk/signature-v4-crt

Because aws-crt contains platform-specific binary code, the following use cases are not supported in MRAP:

  • browsers
  • React Native
  • bundle as Node.js application

The suggested use case for supporting MRAP in Node.js app is to deploy your application with the complete node_modules folder.

Describe alternatives you've considered

We are working on solutions to unblock the MRAP for all the supported platforms.

Related issues:

oskbor commented 7 months ago

So I have spent the last weeks setting up buckets, replication rules, an Multi Region Access Point and I have gotten STS to sign an access token that I have now passed to my client. And here I am, just now finding out that oh btw, multi region access points cannot be queried from the browser. Que? I'm fuming right now.

Isn't that like a very common use case to query s3 from the browser? At least put this information in some MRAP docs 😢 Very sad to have spent so much time on this.

Please tell me that there is a way to do GetObject from a browser via an mrap.

zhuming commented 6 months ago

anywhere I can find a sample code or documentation how to use @aws-sdk/signature-v4-crt to sign an http request sent to aws service. I'm using aws opensearch client, but it has a gap in covering all apis supported by opensearch (elasticSearch), compared to elasticSearchClient. So I can either switch to elasticSearch client, or keep using openSearchClient, but with a polyfill of sending a few signed http request directly to opensearch service. Either way, I need to work on this signing request. I used to use v2 to sign elasticSearch client request, but I believe it's deprecated now. Any suggestions?