search

thumbor-community / aws

Thumbor AWS extensions
MIT License
155 stars 70 forks source link

Fallback to http_loader if s3 loader use with s3 loader #127

Closed srettha closed 6 years ago

srettha commented 6 years ago

Hi, I'm currently testing tc_aws.loaders.s3_loader. I tested with default loader thumbor.loaders.http_loader everything works as expect. But after I switched to s3_loader, it will only look for an image in the bucket, without try grabbing it from url anymore.

Stacktrace

Noted that I changed some of response metadata

thumbor:WARNING ERROR retrieving image from S3 images/www.brandwatch.com/wp-content/resize/uploads/brandwatch/troll.jpg: 
{
  'ResponseMetadata': {
    'HTTPStatusCode': 404,
    'HostId': 'abcdefg',
    'RequestId': 'abcdefg',
    'HTTPHeaders': {
      'X-Amz-Id-2': 'abcdefg',
      'X-Http-Reason': 'Not Found',
      'Server': 'AmazonS3',
      'Transfer-Encoding': 'chunked',
      'X-Amz-Request-Id': 'abcdefg',
      'Date': 'Sat, 06 Oct 2018 06:29:44 GMT',
      'Content-Type': 'application/xml'
    }
},
'Error': {
  'Message': 'The specified key does not exist.',
  'Code': 'NoSuchKey',
  'Key': 'images/www.brandwatch.com/wp-content/resize/uploads/brandwatch/troll.jpg'
  }
}
tornado.access:WARNING 404 GET /EdyZzcRAx3tKHrrri9p1r3ycqwM=/100x100/www.brandwatch.com/wp-content/resize/uploads/brandwatch/troll.jpg (100.115.183.85) 496.16ms

Versions

from cloudposse repo, here is theirs pip.txt

argparse==1.4.0
envtpl==0.5.0
boto==2.39.0
dateutils==0.6.6
numpy==1.11.0
pyremotecv==0.5.0
remotecv==2.2.1
opencv-engine==1.0.1
redis==2.10.5
thumbor==6.0.1
tc_redis==1.0.1
tc_aws==6.0.1
tc_core==0.3.0
tc_shortener==0.2.1
raven==5.12.0

Configuration

I install with cloudposse/thumbor chart

LOADER = 'tc_aws.loaders.s3_loader'
RESULT_STORAGE = 'tc_aws.result_storages.s3_storage'
UPLOAD_ENABLED = True
UPLOAD_PHOTO_STORAGE = 'tc_aws.storages.s3_storage'

############################## TC_AWS ##########################################

TC_AWS_REGION = 'ap-southeast-1' # AWS Region
TC_AWS_STORAGE_BUCKET = 'staging-thumbor' # S3 bucket for Storage
TC_AWS_STORAGE_ROOT_PATH = 'images' # S3 path prefix for Storage bucket
TC_AWS_LOADER_BUCKET = 'staging-thumbor' #S3 bucket for loader
TC_AWS_LOADER_ROOT_PATH = 'images' # S3 path prefix for Loader bucket
TC_AWS_RESULT_STORAGE_BUCKET = 'staging-thumbor' # S3 bucket for result Storage
TC_AWS_RESULT_STORAGE_ROOT_PATH = 'results' # S3 path prefix for Result storage bucket
TC_AWS_STORAGE_SSE = False
TC_AWS_STORAGE_RRS = False
TC_AWS_ENABLE_HTTP_LOADER = True
TC_AWS_ALLOWED_BUCKETS = False # List of allowed bucket to be requested
TC_AWS_STORE_METADATA = False # Store result with metadata (for instance content-type)

Steps to reproduce

  1. curl thumbor-service with image url 
    curl -I -X GET http://localhost:8080/content/thumbs/EdyZzcRAx3tKHrrri9p1r3ycqwM=/100x100/www.brandwatch.com/wp-content/resize/uploads/brandwatch/troll.jpg
Bladrak commented 6 years ago

Hello and thanks for reporting your issue. Loader fallback would be more on the Thumbor's project scope, as we only handle AWS connectivity on our side. You can also implement multiple thumbor processes, each with its own configuration (ie loader) with a front proxy to redirect to the correct thumbor instance based on the URL (which is what we do most of the time). Thanks again, feel free to ask any question.