S3StorageURL Authorization header is malformed; the region 'auto' is wrong.

Sam-tesouro commented 3 months ago

Component(s)

controlplane

Component version

controlplane@0.97.1

wgc version

wgc@0.58.5

controlplane version

controlplane@0.97.1

router version

N/A

What happened?

Description

Targeting s3 for CDN and ControlPlane with URL format https://ID:SECRET@s3.[REGION].amazonaws.com/[BUCKET-NAME] leads to

"stack":"AuthorizationHeaderMalformed: The authorization header is malformed; the region 'auto' is wrong; expecting '[REGION]'"

Steps to Reproduce

https://github.com/Sam-tesouro/cosmo-controlplane-s3-url-issue/tree/main

Expected Result

Successfully list buckets

Actual Result

node_modules/@smithy/smithy-client/dist-cjs/index.js:839
  const response = new exceptionCtor({
                   ^
AuthorizationHeaderMalformed: The authorization header is malformed; the region 'auto' is wrong; expecting 'us-east-1'
    at throwDefaultError

Environment information

Environment

OS: (e.g., "Ubuntu 20.04") Package Manager: pnpm, npm, yarn, etc Compiler(if manually compiled): (e.g., "go 14.2")

Router configuration

No response

Router execution config

No response

Log output

No response

Additional context

The s3 client is forcing path style urls which were originally slated for deprecation 4 years ago. https://aws.amazon.com/blogs/aws/amazon-s3-path-deprecation-plan-the-rest-of-the-story/

Is there any reason why you aren't supporting Virtual-hosted–style requests?

Regardless I suspect I am just missing the correct incantation for an s3 storage url for Cosmo. After trying many different permutations I am hoping it's simple and I can save others the headache in the future with this issue!

Thank you!

github-actions[bot] commented 3 months ago

WunderGraph commits fully to Open Source and we want to make sure that we can help you as fast as possible. The roadmap is driven by our customers and we have to prioritize issues that are important to them. You can influence the priority by becoming a customer. Please contact us here.

jensneuse commented 3 months ago

We're taking a look, thank you!

AndreasZeissner commented 2 months ago

Hi @Sam-tesouro,

I wanted to let you know that we’ve recently added support for AWS S3 URLs.

You might want to have a look here:

helm-cosmo@v0.11.0:

GitHub Release
ArtifactHub Package Here’s an example for CDN configuration:

cdn: configuration: s3StorageUrl: 'https://[username]:[password]@your-bucket.s3.amazonaws.com' s3Region: 'us-east-1'


You can find more options here:

- [Values.yaml - Line 120](https://github.com/wundergraph/cosmo/blob/d8485e0a88e63f7201c246af65d6a21e736ac7af/helm/cosmo/values.yaml#L120)
- [Values.yaml - Line 196](https://github.com/wundergraph/cosmo/blob/d8485e0a88e63f7201c246af65d6a21e736ac7af/helm/cosmo/values.yaml#L196)

In these sections, you'll find details on how to pass secret keys, access key IDs, and more. The CDN configuration needs to be applied similarly for the controlplane. 

Give it a try and see how it works for your setup.

Additional release links:

- controlplane@0.105.0: [GitHub Release](https://github.com/wundergraph/cosmo/releases/tag/controlplane%400.105.0)
- cdn@0.10.0: [GitHub Release](https://github.com/wundergraph/cosmo/releases/tag/cdn%400.10.0)

AndreasZeissner commented 2 months ago

Hi @Sam-tesouro,

we had to ship a small bug fix: https://github.com/wundergraph/cosmo/releases/tag/controlplane%400.105.1

You would need to set you configuration like so:

cdn:
  configuration:
    s3StorageUrl: 'https://[username]:[password]@your-bucket.s3.amazonaws.com'
    s3Region: 'us-east-1'
    s3ForcePathStyle: 'false'

when updating to this version, let me know how things work for you.

Thanks!

wundergraph / cosmo