search

protomaps / go-pmtiles

Single-file executable tool for working with PMTiles archives
BSD 3-Clause "New" or "Revised" License
356 stars 49 forks source link

s3-like compatibility #112

Closed zifeo closed 8 months ago

zifeo commented 8 months ago

while gocloud seems to support s3ForcePathStyle=true, it does not seem possible to use it with pmtiles:

2024/01/04 00:45:10 main.go:128: Failed to create new server, open bucket s3://REDACTED?awssdk=v2&endpoint=REDACTED&region=us-east-1&s3ForcePathStyle=true: unknown query parameter "s3ForcePathStyle"
bdon commented 8 months ago

If you are inputting the URLs can you use virtual host-style URLs instead? AWS has been trying to deprecate path style URLs for a while https://docs.aws.amazon.com/AmazonS3/latest/userguide/VirtualHosting.html#path-style-access so it seem like they should be 1:1 equivalent.

zifeo commented 8 months ago

@bdon Thanks for getting back, unfortunately our storage provider does not offer virtual ones, only path style are supported.

bdon commented 8 months ago

Which storage provider or storage engine is this so others can reproduce?

zifeo commented 8 months ago

@bdon Openstack Swift, but I suppose you can also use minio to replicate.

bdon commented 8 months ago

PR here: https://github.com/protomaps/go-pmtiles/pull/113

Currently we force the use of awssdkv2, but v1 is necessary to support those flags and work with Minio. Defaulting to v1 shouldn't have any functional impact. The alternative is to add more storage implementation-specific logic to go-pmtiles but that would defeat the purpose of using the gocloud abstraction.

Aligns with suggested way of using Minio in gocloud docs: https://gocloud.dev/howto/blob/#s3-compatible

bdon commented 8 months ago

I've merged it into main, please see if it fixes the issue and I'll make a minor release.

zifeo commented 8 months ago

@bdon confirmed, thanks a lot!

bdon commented 8 months ago

Released in https://github.com/protomaps/go-pmtiles/releases/tag/v1.12.0

bdon commented 1 week ago

FYI, AWS is putting aws-sdk-go v1 into maintenance mode; v2 will be the default going forward: https://github.com/google/go-cloud/releases/tag/v0.39.0

For the next minor version I am bumping the gocloud version

zifeo commented 1 week ago

@bdon Thanks for letting me know, what is the expect change? It seems unclear to me

bdon commented 1 week ago

Based on those release notes Gocloud will be removing aws-sdk-v1 in early 2025.

The AWS SDK V2 definitely supports forcing path-style S3 buckets, not quite sure how it's exposed through gocloud S3 connection strings.

zifeo commented 1 week ago

Seems to be the same: https://github.com/google/go-cloud/blob/master/aws/aws.go#L99