S3 virtual-style addressing for non AWS endpoints

[ebabani]

Any consideration to allowing DNS style bucked addresses for non s3 endpoints?

Expected Behavior

Expect virtual style addressing to work for non-s3 endpoints as well.

Current Behavior

Currently DNS style addressing is only enabled if the endpoint starts with s3. (https://github.com/aws/aws-sdk-java-v2/blob/1203086f21355a1fba45d4f0119b090cb3d77995/services/s3/src/main/java/software/amazon/awssdk/services/s3/handlers/EndpointAddressInterceptor.java#L141)

There are s3 compatible providers that support only DNS style addressing, which makes this SDK harder to use with them. (https://www.alibabacloud.com/help/doc-detail/64919.htm)

Possible Solution

Remove the restriction on DNS style addresses only working for the AWS s3 endpoints.

Context

Trying to use this client on Alicloud OSS.

Your Environment

<N/A>

[ebabani]

The change is fairly straight forward to make and I can send a PR, but just want to know what the direction is regarding this. At the moment if feels like for s3 compatible providers this sdk leans towards forcing the path style.

[millems]

It's tough for us to test to make sure we're compatible with all of them, but we're definitely not against it. We'd gladly accept a PR that fixes the issue.

[blackstorm]

i decide back to sdk v1. Sad!

[yannrouillard]

Currently experiencing this issue as well.

@millems I saw you were not keen to merge the PR https://github.com/aws/aws-sdk-java-v2/pull/763, is there some other way to make it work that you would accept like creating a new option to force virtual-style addressing?

[millems]

It's going to be difficult to figure out how to support this, with new features like S3 access points adding even more complexity to the URL building process, and the S3 specification being implemented differently by different providers.

Is there a specific non-AWS runtime we can focus in on supporting, so that we don't just have it be "all non-AWS S3 implementations"?

[yannrouillard]

Hi @millems,

I am not sure I am legimitate to say which non-AWS runtime should be better to focus on. You could target Google / Alicloud implementations to facilite migrations from them to AWS but I would find weird that you have to follow Google / Alicloud modifications to stay compatible (and pay some complexities price on the way). (In our case it is with Alicloud implementation that we have an issue with using this library)

Supporting an alternate implementation like Minio sounds better as it is an open-source project and can be used by people to use S3-like interfaces on-premise in hybrid configuration (or for testing purpose).

In the end anyway, I don't see how you can maintain compatibility with existing behaviour without introducing a new option or a new configuration.

I saw you were refactoring to separate between accessPointEndpointResolver and bucketEndointResolver, maybe you could allow to set a custom resolver so that library users have always a way to customize / override behavior like this without incurring too high maintainance on your side.

In all cases, it would be nice to document this behaviour (_virtualAccesStyle can't be enabled for endpoints that do not begin by s3) in the official documentation to avoid the surprise effect. Other SDK don't have this behaviour AFAIK so it took us some time to find the cause.

[yasminetalby]

Hello all,

This feature-request has been implemented as part of the 2.18.x version release.

As per the release Changelog, the new behavior is as follow:

When including an endpoint override, the SDK will use virtual bucket addressing as long as the bucket is a valid DNS label. This is different from the previous behavior where this only happens if the endpoint override begins with s3 : see changelog

As a side-note, if your bucket is not a valid DNS label , the behavior would be to not use virtual bucket addressing.

Thank you very much for your feature-request submission and feedback!

Best,

Yasmine

[github-actions[bot]]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see. If you need more assistance, please open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.