Auto-detect bucket region

[pdecat]

This change allows to auto-detect the bucket's region instead of having to define the region in the configuration files or the AWS_DEFAULT_REGION environment variable.

[omribahumi]

Thanks for the PR! I'm not sure I understand why this is needed. I don't have a default region defined and I can run the tool on buckets in us-east-1 and ap-northeast-1 without modifying the code. I think that specifying a region for S3 client is only for creating buckets. Because there's just one global S3 bucket namespace, the region can be auto detected when accessing them.

Can you try reproducing your issue without having a default region defined?

[pdecat]

Hi Omri,

I don't have any default region in my ~/.aws/config.

Without that change, I get the following error for buckets in the eu-west-1, eu-central-1 and ap-southeast-1 regions when the AWS_DEFAULT_REGION environment variable is not set:

$ ncdu-s3 s3://mybucket-in-eu-west-1 mybucket-in-eu-west-1.json                                                                                               
Traceback (most recent call last):
  File "/home/pdecat/workspaces/thirdparty/ncdu-s3/venv-ncdu-s3/bin/ncdu-s3", line 9, in <module>
    load_entry_point('ncdu-s3==0.2', 'console_scripts', 'ncdu-s3')()
  File "/home/pdecat/workspaces/thirdparty/ncdu-s3/venv-ncdu-s3/local/lib/python2.7/site-packages/click-5.1-py2.7.egg/click/core.py", line 700, in __call__
    return self.main(*args, **kwargs)
  File "/home/pdecat/workspaces/thirdparty/ncdu-s3/venv-ncdu-s3/local/lib/python2.7/site-packages/click-5.1-py2.7.egg/click/core.py", line 680, in main
    rv = self.invoke(ctx)
  File "/home/pdecat/workspaces/thirdparty/ncdu-s3/venv-ncdu-s3/local/lib/python2.7/site-packages/click-5.1-py2.7.egg/click/core.py", line 873, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/pdecat/workspaces/thirdparty/ncdu-s3/venv-ncdu-s3/local/lib/python2.7/site-packages/click-5.1-py2.7.egg/click/core.py", line 508, in invoke
    return callback(*args, **kwargs)
  File "/home/pdecat/workspaces/thirdparty/ncdu-s3/venv-ncdu-s3/local/lib/python2.7/site-packages/click-5.1-py2.7.egg/click/decorators.py", line 16, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "build/bdist.linux-x86_64/egg/ncdu_s3/main.py", line 25, in main
  File "build/bdist.linux-x86_64/egg/ncdu_s3/s3_directory_generator.py", line 29, in generator
  File "/home/pdecat/workspaces/thirdparty/ncdu-s3/venv-ncdu-s3/local/lib/python2.7/site-packages/boto3-1.1.3-py2.7.egg/boto3/resources/collection.py", line 82, in __iter__
    for page in self.pages():
  File "/home/pdecat/workspaces/thirdparty/ncdu-s3/venv-ncdu-s3/local/lib/python2.7/site-packages/boto3-1.1.3-py2.7.egg/boto3/resources/collection.py", line 165, in pages
    for page in pages:
  File "/home/pdecat/workspaces/thirdparty/ncdu-s3/venv-ncdu-s3/local/lib/python2.7/site-packages/botocore-1.2.3-py2.7.egg/botocore/paginate.py", line 83, in __iter__
    response = self._make_request(current_kwargs)
  File "/home/pdecat/workspaces/thirdparty/ncdu-s3/venv-ncdu-s3/local/lib/python2.7/site-packages/botocore-1.2.3-py2.7.egg/botocore/paginate.py", line 155, in _make_request
    return self._method(**current_kwargs)
  File "/home/pdecat/workspaces/thirdparty/ncdu-s3/venv-ncdu-s3/local/lib/python2.7/site-packages/botocore-1.2.3-py2.7.egg/botocore/client.py", line 268, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/home/pdecat/workspaces/thirdparty/ncdu-s3/venv-ncdu-s3/local/lib/python2.7/site-packages/botocore-1.2.3-py2.7.egg/botocore/client.py", line 333, in _make_api_call
    raise ClientError(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (PermanentRedirect) when calling the ListObjects operation: The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint.

It is indeed not required for buckets in the us-east-1 region.

[omribahumi]

@pdecat what's your boto3 version?

[pdecat]

It's from a fresh virtualenv:

$ pip freeze | grep boto
boto3==1.1.3
botocore==1.2.3

[omribahumi]

@pdecat any chance your bucket name has dots in it? :)

[pdecat]

Yep, they all do.

[omribahumi]

@pdecat thanks!

[pdecat]

You're welcome, thanks for your work!