Error when trying to get image from bucket

[paulux84]

Hi, i have this image and i can normally get it whit my thumbor service here http://thumborservice-weflymate.rhcloud.com/unsafe/https://s3-eu-west-1.amazonaws.com/3cube/thumbor/storage/035de4821bbb4a109a8cfcb2bf4af3a08f9ee298. As you can see the image is on aws s3 and i have configured my thumbor instance to access this s3 instance, but if i try to get image directly from instance (url) i get the following error:

2016-11-17 14:08:31 thumbor:ERROR ERROR: Traceback (most recent call last):
  File "/var/lib/openshift/58107eca2d5271d5a10000da/app-root/data/lib/python2.7/site-packages/thumbor/handlers/__init__.py", line 142, in get_image
    self.context.request.image_url
  File "/var/lib/openshift/58107eca2d5271d5a10000da/app-root/data/lib/python2.7/site-packages/tornado/gen.py", line 1015, in run
    value = future.result()
  File "/var/lib/openshift/58107eca2d5271d5a10000da/app-root/data/lib/python2.7/site-packages/tornado/concurrent.py", line 237, in result
    raise_exc_info(self._exc_info)
  File "/var/lib/openshift/58107eca2d5271d5a10000da/app-root/data/lib/python2.7/site-packages/tornado/gen.py", line 1024, in run
    yielded = self.gen.send(value)
  File "/var/lib/openshift/58107eca2d5271d5a10000da/app-root/data/lib/python2.7/site-packages/thumbor/handlers/__init__.py", line 516, in _fetch
    loader_result = yield self.context.modules.loader.load(self.context, url)
  File "/var/lib/openshift/58107eca2d5271d5a10000da/app-root/data/lib/python2.7/site-packages/tornado/concurrent.py", line 478, in wrapper
    future.result()
  File "/var/lib/openshift/58107eca2d5271d5a10000da/app-root/data/lib/python2.7/site-packages/tornado/concurrent.py", line 237, in result
    raise_exc_info(self._exc_info)
  File "/var/lib/openshift/58107eca2d5271d5a10000da/app-root/data/lib/python2.7/site-packages/tornado/concurrent.py", line 466, in wrapper
    result = f(*args, **kwargs)
  File "/var/lib/openshift/58107eca2d5271d5a10000da/app-root/data/lib/python2.7/site-packages/tc_aws/loaders/s3_loader.py", line 49, in load
    bucket_loader.get(key, callback=handle_data)
  File "/var/lib/openshift/58107eca2d5271d5a10000da/app-root/data/lib/python2.7/site-packages/tornado/concurrent.py", line 478, in wrapper
    future.result()
  File "/var/lib/openshift/58107eca2d5271d5a10000da/app-root/data/lib/python2.7/site-packages/tornado/concurrent.py", line 237, in result
    raise_exc_info(self._exc_info)
  File "/var/lib/openshift/58107eca2d5271d5a10000da/app-root/data/lib/python2.7/site-packages/tornado/concurrent.py", line 466, in wrapper
    result = f(*args, **kwargs)
  File "/var/lib/openshift/58107eca2d5271d5a10000da/app-root/data/lib/python2.7/site-packages/tc_aws/aws/bucket.py", line 48, in get
    Key=self._clean_key(path),
  File "/var/lib/openshift/58107eca2d5271d5a10000da/app-root/data/lib/python2.7/site-packages/tornado_botocore/base.py", line 97, in call
    return self._make_api_call(operation_name=self.operation, api_params=kwargs, callback=callback)
  File "/var/lib/openshift/58107eca2d5271d5a10000da/app-root/data/lib/python2.7/site-packages/tornado_botocore/base.py", line 58, in _make_api_call
    request_dict = self.client._convert_to_request_dict(api_params, operation_model)
  File "/var/lib/openshift/58107eca2d5271d5a10000da/app-root/data/lib/python2.7/site-packages/botocore/client.py", line 563, in _convert_to_request_dict
    params=api_params, model=operation_model, context=context)
  File "/var/lib/openshift/58107eca2d5271d5a10000da/app-root/data/lib/python2.7/site-packages/botocore/hooks.py", line 227, in emit
    return self._emit(event_name, kwargs)
  File "/var/lib/openshift/58107eca2d5271d5a10000da/app-root/data/lib/python2.7/site-packages/botocore/hooks.py", line 210, in _emit
    response = handler(**kwargs)
  File "/var/lib/openshift/58107eca2d5271d5a10000da/app-root/data/lib/python2.7/site-packages/botocore/utils.py", line 947, in redirect_from_cache
    context['signing'] = {'bucket': bucket}
TypeError: 'NoneType' object does not support item assignment

2016-11-17 14:08:31 thumbor:ERROR [BaseHandler] get_image failed for url `035de4821bbb4a109a8cfcb2bf4af3a08f9ee298`. error: `'NoneType' object does not support item assignment`
2016-11-17 14:08:31 tornado.access:ERROR 500 GET /unsafe/100x100/035de4821bbb4a109a8cfcb2bf4af3a08f9ee298 (127.3.209.129) 1395.57ms

Below there a part of my thumbor.conf file:

## The loader thumbor should use to load the original image. This must be the
## full name of a python module (python must be able to import it)
## Defaults to: thumbor.loaders.http_loader
#LOADER = 'thumbor.loaders.http_loader'
#LOADER = 'tc_aws.loaders.s3_loader'
LOADER = 'tc_aws.loaders.s3_loader'

## The file storage thumbor should use to store original images. This must be the
## full name of a python module (python must be able to import it)
## Defaults to: thumbor.storages.file_storage
STORAGE = 'thumbor_aws.storages.s3_storage'
AWS_ACCESS_KEY = 'KEY'
AWS_SECRET_KEY = 'SECRET'
STORAGE_BUCKET = '3cube'

## The result storage thumbor should use to store generated images. This must be
## the full name of a python module (python must be able to import it)
## Defaults to: None
RESULT_STORAGE = 'thumbor_aws.result_storages.s3_storage'
RESULT_STORAGE_BUCKET = '3cube'

TC_AWS_REGION='eu-west-1' # AWS Region

TC_AWS_STORAGE_BUCKET= '3cube' # S3 bucket for Storage
TC_AWS_STORAGE_ROOT_PATH='' # S3 path prefix for Storage bucket

TC_AWS_LOADER_BUCKET='3cube' #S3 bucket for loader
TC_AWS_LOADER_ROOT_PATH='' # S3 path prefix for Loader bucket

TC_AWS_RESULT_STORAGE_BUCKET= '3cube' # S3 bucket for result Storage
TC_AWS_RESULT_STORAGE_ROOT_PATH='' # S3 path prefix for Result storage bucket

# put data into S3 using the Server Side Encryption functionality to
# encrypt data at rest in S3
# https://aws.amazon.com/about-aws/whats-new/2011/10/04/amazon-s3-announces-server-side-encryption-support/
TC_AWS_STORAGE_SSE=False

# put data into S3 with Reduced Redundancy
# https://aws.amazon.com/about-aws/whats-new/2010/05/19/announcing-amazon-s3-reduced-redundancy-storage/
TC_AWS_STORAGE_RRS=False

# Enable HTTP Loader as well?
# This would allow you to load watermarks in over your images dynamically through a URI
# E.g.
# http://your-thumbor.com/unsafe/filters:watermark(http://example.com/watermark.png,0,0,50)/s3_bucket/photo.jpg
TC_AWS_ENABLE_HTTP_LOADER=True

TC_AWS_ALLOWED_BUCKETS=False # List of allowed bucket to be requested
TC_AWS_STORE_METADATA=False # Store result with metadata (for instance content-type)

## The imaging engine thumbor should use to perform image operations. This must
## be the full name of a python module (python must be able to import it)
## Defaults to: thumbor.engines.pil
ENGINE = 'thumbor.engines.pil'

Thanks for help ;)

[aoqfonseca]

@paulux84

I think this is an issue with botocore version. We are working on this at now but we don't have an answer for that.

For now, You can use a specific version of botocore: pip install botocore==1.3.7

Sorry about this problem. As i wrote above, we work at this. Look issue #62

[paulux84]

Whit botocore 1.3.7 i get the following

2016-11-18 13:11:47 thumbor:ERROR ERROR: Traceback (most recent call last):
  File "/var/lib/openshift/58107eca2d5271d5a10000da/app-root/data/lib/python2.7/site-packages/thumbor/handlers/__init__.py", line 142, in get_image
    self.context.request.image_url
  File "/var/lib/openshift/58107eca2d5271d5a10000da/app-root/data/lib/python2.7/site-packages/tornado/gen.py", line 1015, in run
    value = future.result()
  File "/var/lib/openshift/58107eca2d5271d5a10000da/app-root/data/lib/python2.7/site-packages/tornado/concurrent.py", line 237, in result
    raise_exc_info(self._exc_info)
  File "/var/lib/openshift/58107eca2d5271d5a10000da/app-root/data/lib/python2.7/site-packages/tornado/gen.py", line 1024, in run
    yielded = self.gen.send(value)
  File "/var/lib/openshift/58107eca2d5271d5a10000da/app-root/data/lib/python2.7/site-packages/thumbor/handlers/__init__.py", line 516, in _fetch
    loader_result = yield self.context.modules.loader.load(self.context, url)
  File "/var/lib/openshift/58107eca2d5271d5a10000da/app-root/data/lib/python2.7/site-packages/tornado/concurrent.py", line 478, in wrapper
    future.result()
  File "/var/lib/openshift/58107eca2d5271d5a10000da/app-root/data/lib/python2.7/site-packages/tornado/concurrent.py", line 237, in result
    raise_exc_info(self._exc_info)
  File "/var/lib/openshift/58107eca2d5271d5a10000da/app-root/data/lib/python2.7/site-packages/tornado/concurrent.py", line 466, in wrapper
    result = f(*args, **kwargs)
  File "/var/lib/openshift/58107eca2d5271d5a10000da/app-root/data/lib/python2.7/site-packages/tc_aws/loaders/s3_loader.py", line 49, in load
    bucket_loader.get(key, callback=handle_data)
  File "/var/lib/openshift/58107eca2d5271d5a10000da/app-root/data/lib/python2.7/site-packages/tornado/concurrent.py", line 478, in wrapper
    future.result()
  File "/var/lib/openshift/58107eca2d5271d5a10000da/app-root/data/lib/python2.7/site-packages/tornado/concurrent.py", line 237, in result
    raise_exc_info(self._exc_info)
  File "/var/lib/openshift/58107eca2d5271d5a10000da/app-root/data/lib/python2.7/site-packages/tornado/concurrent.py", line 466, in wrapper
    result = f(*args, **kwargs)
  File "/var/lib/openshift/58107eca2d5271d5a10000da/app-root/data/lib/python2.7/site-packages/tc_aws/aws/bucket.py", line 48, in get
    Key=self._clean_key(path),
  File "/var/lib/openshift/58107eca2d5271d5a10000da/app-root/data/lib/python2.7/site-packages/tornado_botocore/base.py", line 97, in call
    return self._make_api_call(operation_name=self.operation, api_params=kwargs, callback=callback)
  File "/var/lib/openshift/58107eca2d5271d5a10000da/app-root/data/lib/python2.7/site-packages/tornado_botocore/base.py", line 60, in _make_api_call
    operation_model=operation_model, request_dict=request_dict, callback=callback)
  File "/var/lib/openshift/58107eca2d5271d5a10000da/app-root/data/lib/python2.7/site-packages/tornado_botocore/base.py", line 54, in _make_request
    request_dict=request_dict, operation_model=operation_model, callback=callback)
  File "/var/lib/openshift/58107eca2d5271d5a10000da/app-root/data/lib/python2.7/site-packages/tornado_botocore/base.py", line 32, in _send_request
    request = self.endpoint.create_request(request_dict, operation_model)
  File "/var/lib/openshift/58107eca2d5271d5a10000da/app-root/data/lib/python2.7/site-packages/botocore/endpoint.py", line 121, in create_request
    operation_name=operation_model.name)
  File "/var/lib/openshift/58107eca2d5271d5a10000da/app-root/data/lib/python2.7/site-packages/botocore/hooks.py", line 226, in emit
    return self._emit(event_name, kwargs)
  File "/var/lib/openshift/58107eca2d5271d5a10000da/app-root/data/lib/python2.7/site-packages/botocore/hooks.py", line 209, in _emit
    response = handler(**kwargs)
  File "/var/lib/openshift/58107eca2d5271d5a10000da/app-root/data/lib/python2.7/site-packages/botocore/signers.py", line 81, in handler
    return self.sign(operation_name, request)
  File "/var/lib/openshift/58107eca2d5271d5a10000da/app-root/data/lib/python2.7/site-packages/botocore/signers.py", line 116, in sign
    signer.add_auth(request=request)
  File "/var/lib/openshift/58107eca2d5271d5a10000da/app-root/data/lib/python2.7/site-packages/botocore/auth.py", line 621, in add_auth
    raise NoCredentialsError
NoCredentialsError: Unable to locate credentials

[aoqfonseca]

@paulux84

Please take a look at botocore docs (https://boto3.readthedocs.io/en/latest/guide/quickstart.html#configuration) and you'll see how you configure to access S3.

You need to create a aws credential file to access with your SECRET and KEY .

[paulux84]

with boto 1.4.7 all works...with 1.3.7 i have created an external aws config file with credential and set its location throught AWS_CONFIG_FILE environment variable...now all works

[paulux84]

Thanks for all;)

[aoqfonseca]

@paulux84

I looked at botocore repository and i believe version 1.4.76 of Botocore fixed this issue. We will try here but you can try for yourself.

[paulux84]

I will try...thanks

[johntiror]

still the same issue up to botocore 1.4.81

[aoqfonseca]

Tks!

[johntiror]

Ok, but how can I solve? With botocore 1.4.7 I reveive the error "Unable to locate credentials" even if I followed the boto docs and I run "aws configure". Could you please explain me how to locate the credentials? I read about create a AWS_CONFIG_FILE environment variable but maybe I did something wrong but it's not working, always same error.

I created the environment variable like this: EXPORT AWS_CONFIG_FILE=/home/ec2-user/.aws/credentials

[aoqfonseca]

@johntiror Did you create AWS_CONFIG_FILE env variable? Perhaps you should create this variables with same user that you use to run your python process.

[johntiror]

yes, you were right. Thanks very much!