recent versions of boto3/botocore cause mysterious errors

[r-b-g-b]

I was getting this strange error upon installing cottoncandy in a fresh virtualenv. Downgrading boto3-->boto3==1.3.1 (which also downgrades botocore==1.5.86-->botocore==1.4.93) fixed it. I'll try to look into why when I get a chance. But that sucks.

In [2]: cci = get_cc_interface()
---------------------------------------------------------------------------
ClientError                               Traceback (most recent call last)
<ipython-input-2-f5fa24cc8fda> in <module>()
----> 1 cci = get_cc_interface('robertg_sorcery')

/auto/k1/robertg/code/glabtools/glabtools/io.py in get_cc_interface(bucket_name, force_bucket_creation, verbose)
     95             return answer
     96 
---> 97     cci = GLabCC(bucket_name=bucket_name, verbose=verbose)
     98     return cci
     99 

/auto/k1/robertg/code/glabtools/glabtools/io.py in __init__(self, bucket_name, ACCESS_KEY, SECRET_KEY, url, force_bucket_creation, verbose)
     50                                          url=url,
     51                                          force_bucket_creation=force_bucket_creation,
---> 52                                          verbose=verbose)
     53 
     54         def create_bucket(self, bucket_name):

/auto/k1/robertg/code/cottoncandy/cottoncandy/interfaces.py in __init__(self, *args, **kwargs)
    926             Cottoncandy interface object
    927         """
--> 928         super(FileSystemInterface, self).__init__(*args, **kwargs)
    929 
    930     def lsdir(self, path='/', limit=10**3):

/auto/k1/robertg/code/cottoncandy/cottoncandy/interfaces.py in __init__(self, *args, **kwargs)
    483             Cottoncandy interface object
    484         """
--> 485         super(ArrayInterface, self).__init__(*args, **kwargs)
    486 
    487     @clean_object_name

/auto/k1/robertg/code/cottoncandy/cottoncandy/interfaces.py in __init__(self, bucket_name, ACCESS_KEY, SECRET_KEY, url, force_bucket_creation, verbose, backe
nd, **kwargs)
     94             self.backend_interface = S3Client(bucket_name, ACCESS_KEY, SECRET_KEY, url,
     95                                               force_bucket_creation,
---> 96                                               **kwargs)
     97         elif backend == 'gdrive':
     98             from .gdriveclient import GDriveClient

/auto/k1/robertg/code/cottoncandy/cottoncandy/s3client.py in __init__(self, bucket, access_key, secret_key, s3url, force_bucket_creation, **kwargs)
     65         self.connection = S3Client.connect(access_key, secret_key, s3url, **kwargs)
     66         self.url = s3url
---> 67         if self.check_bucket_exists(bucket):
     68             self.set_current_bucket(bucket)
     69         else:

/auto/k1/robertg/code/cottoncandy/cottoncandy/s3client.py in check_bucket_exists(self, bucket_name)
    139                 exists = False
    140             else:
--> 141                 raise e
    142         else:
    143             exists = True

/auto/k1/robertg/code/cottoncandy/cottoncandy/s3client.py in check_bucket_exists(self, bucket_name)
    134         """
    135         try:
--> 136             self.connection.meta.client.head_bucket(Bucket = bucket_name)
    137         except botocore.exceptions.ClientError as e:
    138             if e.response['Error']['Code'] == "404":

~/virtualenvs/science3/lib/python3.5/site-packages/botocore/client.py in _api_call(self, *args, **kwargs)
    308                     "%s() only accepts keyword arguments." % py_operation_name)
    309             # The "self" in this scope is referring to the BaseClient.
--> 310             return self._make_api_call(operation_name, kwargs)
    311 
    312         _api_call.__name__ = str(py_operation_name)

~/virtualenvs/science3/lib/python3.5/site-packages/botocore/client.py in _make_api_call(self, operation_name, api_params)
    597             error_code = parsed_response.get("Error", {}).get("Code")
    598             error_class = self.exceptions.from_code(error_code)
--> 599             raise error_class(parsed_response, operation_name)
    600         else:
    601             return parsed_response

ClientError: An error occurred (400) when calling the HeadBucket operation: Bad Request

[r-b-g-b]

this reproduces

$ virtualenv boto3_test -p python2
$ source boto3_test/bin/activate
$ git clone https://github.com/gallantlab/cottoncandy
$ cd cottoncandy
$ pip install -r requirements.txt
## installs boto3-1.4.4 botocore-1.5.86
$ pip install .
$ python
>> import cottoncandy as cc
>> cc.get_interface('robertg_sorcery')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "cottoncandy/__init__.py", line 72, in get_interface
    **kwargs)
  File "cottoncandy/interfaces.py", line 1308, in __init__
    super(DefaultInterface, self).__init__(*args, **kwargs)
  File "cottoncandy/interfaces.py", line 928, in __init__
    super(FileSystemInterface, self).__init__(*args, **kwargs)
  File "cottoncandy/interfaces.py", line 485, in __init__
    super(ArrayInterface, self).__init__(*args, **kwargs)
  File "cottoncandy/interfaces.py", line 96, in __init__
    **kwargs)
  File "cottoncandy/s3client.py", line 67, in __init__
    if self.check_bucket_exists(bucket):
  File "cottoncandy/s3client.py", line 141, in check_bucket_exists
    raise e
botocore.exceptions.ClientError: An error occurred (400) when calling the HeadBucket operation: Bad Request

and from here this fixes

>> quit()
$ pip install boto3==1.3.1
...
$ python
>> import cottoncandy as cc
>> cc.get_interface()

that works fine.

[anwarnunez]

Not sure what to make of this. The travis build has that configuration and it's fine (boto3 1.4.4 and botocore 1.5.68): https://travis-ci.org/gallantlab/cottoncandy/jobs/243060408

[r-b-g-b]

Ha botocore==1.5.68 works but 1.5.86 does not. That's lame

[anwarnunez]

interesting, I wonder what has changed here between versions: self.connection.meta.client.head_bucket(Bucket = bucket_name)

[r-b-g-b]

ok to pinpoint this a little better: botocore==1.5.70 works, botocore==1.5.71 fails.

[r-b-g-b]

from botocore CHANGELOG.rst 1.5.71

• enhancement:s3: All S3 requests will now use SigV4 by default so that we can take advantage of stronger security algorithms and improved access key isolation.
• api-change:organizations: Update organizations client to latest version

[anwarnunez]

seems botocore changed the default signature. this issue is therefore likely signature v4 implementation with radosgw / ceph.

the call signature for client.head_bucket(Bucket = bucket_name) might need to be changed to specify additional arguments (signature v2). this can be added to configuration options.

refs: https://github.com/fog/fog/issues/3299 http://tracker.ceph.com/issues/17076 http://tracker.ceph.com/issues/10333

[anwarnunez]

Try this:

import cottoncandy as cc
from botocore.client import Config
config = Config(s3={'addressing_style': 'path'}, signature_version='s3')
cci = cc.get_interface(config=config)

ref: https://github.com/boto/botocore/issues/601

[anwarnunez]

I'm gonna close it b/c it's a combination of user error and radosgw, not boto, botocore.