search

google / go-cloud

The Go Cloud Development Kit (Go CDK): A library and tools for open cloud development in Go.
https://gocloud.dev/
Apache License 2.0
9.57k stars 812 forks source link

blob/s3blob: s3ForcePathStyle stopped working in v0.39.0 #3472

Closed caarlos0 closed 2 weeks ago

caarlos0 commented 3 months ago

Describe the bug

This was flagged by goreleaser's tests when dependabot opened the pr updating from v0.38 to v0.39.

I'm not sure if this is expected and just not documented, or if it is in fact a bug.

To Reproduce

open a bucket (can be a local minio for example) using s3ForcePathStyle.

Expected behavior

To work.

Version

v0.39.0

Additional context

Error:

open bucket s3://basic?endpoint=http%3A%2F%2Flocalhost%3A32812&region=us-east&s3ForcePathStyle=true: unknown query parameter "s3ForcePathStyle"

Also, forcing awssdk=v1 makes it work again.

vangent commented 3 months ago

Please see the release notes (https://github.com/google/go-cloud/releases/tag/v0.39.0). This release changed the default for AWS URLs from their v1 SDK to their v2 SDK. Some parameters are different, s3ForcePathStyle is one that doesn't seem to exist for v2.

You should either update the URL to work with v2 (possibly just dropping that parameter will fix it?), or add awssdk=v1 (which will work for now, but support will be dropped at some point, the release notes proposed 6 months).

caarlos0 commented 3 months ago

yes, I saw the release notes, and also that forcing v1 works, my point was that the specific breaking change is not documented anywhere.

also, just dropping that part of the URL doesn't work (at least not with minio).

vangent commented 3 months ago

OK. Yes, I didn't try to document every specific URL parameter that wouldn't work anymore, just "Most URLs should continue to work, but in some cases you may need to add awssdk=v1 to force V1 explicitly."

vangent commented 3 months ago

just dropping that part of the URL doesn't work (at least not with minio

Please try to figure out what's needed to get V2 to work with Minio (I'm not sure). It's possible we need to add support for more URL parameters for V2.

dmotylev commented 1 month ago

It seems s3ForcePathStyle has been moved to https://github.com/aws/aws-sdk-go-v2/blob/0cbb5aa17f9078cb45dc0e82d3e1d0abee3744a9/service/s3/options.go#L147-L150

vangent commented 1 month ago

Added, see https://github.com/google/go-cloud/pull/3491.

stanhu commented 2 weeks ago

@vangent Would you consider adding an alias for s3ForcePathStyle with use_path_style for backwards compatibility? We've had a few customers hit backup failures due to this URL change after we upgraded to v0.39.0.

vangent commented 2 weeks ago

Sure, that sounds reasonable; will send a PR.

stanhu commented 2 weeks ago

@vangent Thanks so much!