BD2KGenomics / s3am

A fast, parallel, streaming multipart uploader for S3
Other
13 stars 6 forks source link

Add support for buckets in non-US regions #12

Open hannes-ucsc opened 9 years ago

hannes-ucsc commented 9 years ago

By default, S3Connection uses the generic $bucketname.s3.amazonaws.com endpoint which resolves US buckets to the correct US region. However, this does not work for buckets in non-US regions. Either implement a --region command line option to allow users to access non-US buckets or implement logic to follow the redirect that one gets when trying to access a non-US bucket while using the generic endpoint. The latter would be more user-friendly. If a specific region is used for S3 connections, we should use OrdinaryCallingFormat as it allows dots in bucket names without having to hack hostname verification in Python's ssl module as described in https://github.com/boto/boto/issues/2836.

http://docs.aws.amazon.com/general/latest/gr/rande.html#s3_region http://docs.aws.amazon.com/AmazonS3/latest/dev/VirtualHosting.html