search

cube-js / cube

📊 Cube — The Semantic Layer for Building Data Applications
https://cube.dev
Other
17.48k stars 1.74k forks source link

Switch to official AWS SDK for S3 to support IMDSv2 #4041

Open ArthurBaudry opened 2 years ago

ArthurBaudry commented 2 years ago

Is your feature request related to a problem? Please describe. The dependency used for connecting to S3 in cube.js is https://github.com/durch/rust-s3. It only supports IMDSv1 which is insecure and sometimes banned globally for some organisation. IMDSv2 is the secure replacement and isn't supported by rust-s3. As a result, cube.js running in AWS on EC2 (the problem may extend to anything AWS) is unable to access S3 and compute pre-aggregations using S3 since credentials can't be obtained from the instance via IMDS. More documentation on IMDS at https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/configuring-instance-metadata-service.html

Describe the solution you'd like AWS has released their official rust SDK which includes S3 and is in developer preview. It would be great if we could add a feature/config flag to use the AWS SDK over rust-s3 in cube.js. At least while it's in developer preview and ultimately switch to the AWS SDK completely

Describe alternatives you've considered The alternative would be to stick with v1 which might be possible at some organisation. Using other authentication mechanisms such as keypairs is often banned as well.

Additional context Code from rust-s3 getting the credentials from instance metadata is here and you can see it's not requesting a token before issuing the request as recommended with v2

github-actions[bot] commented 2 years ago

If you are interested in working on this issue, please leave a comment below and we will be happy to assign the issue to you. If this is the first time you are contributing a Pull Request to Cube.js, please check our contribution guidelines. You can also post any questions while contributing in the #contributors channel in the Cube.js Slack.