Migrating from old version: 404 on old images

skhro87 commented 4 years ago

we try to migrate from old version 1.1 of tc_aws to new version 6.2.15
all image files are stored in s3
we can upload and retrieve new images to and from s3 with the new server without issues
after we synced all the old files from the old bucket into a the new bucket, we cannot retrieve the old images with the new server
Thumbor logs are showing 404 response from AWS.
using the same SECURITY_KEY and config
we have verified that all images have fully been copied from the old bucket to the new bucket
we have verified that the AWS credentials are correct

The URLs look like this: old: <old_host>/bbvBYK06cBbtxDxWoA3RqrlJHQ8=/50x50/8afd08cb72f146feb704b0506358937e/company_104259_logo_image_url.png new: <new_host>/bbvBYK06cBbtxDxWoA3RqrlJHQ8=/50x50/8afd08cb72f146feb704b0506358937e/company_104259_logo_image_url.png

The strange thing is (I hope I can explain this clearly): When we upload to the servers, and get a link like <host>/<hash1>/<size>/<hash2>/<filename> In the new server, we can find hash2 in our S3 bucket after uploading a new image. In the old server, we can't find hash2 in our S3 bucket after uploading a new image.

Hence we suspect that the old server somehow converts <hash2> when uploading/retrieving, and the new version doesn't. That would explain why we can't load the old images with the new server. However we can't find a setting for this.

Stacktrace

2020-07-24 10:02:43 thumbor:ERROR ERROR retrieving image from S3 
thumbor/storage/8afd08cb72f146feb704b0506358937e/company_104259_logo_image_url.png: 
{
  'ResponseMetadata': {
    'HTTPStatusCode': 404,
    'HostId': 'uihRJ5OSheqQUBRN9X37J8fBl68NBWJVrfTe2thfYXqELCx+fPRkfBcxJ+4+8O76MAgNZimzge4=',
    'RequestId': 'B26B85EC886CB523',
    'HTTPHeaders': {
      'x-amz-id-2': 'uihRJ5OSheqQUBRN9X37J8fBl68NBWJVrfTe2thfYXqELCx+fPRkfBcxJ+4+8O76MAgNZimzge4=',
      'server': 'AmazonS3',
      'transfer-encoding': 'chunked',
      'connection': 'close',
      'x-amz-request-id': 'B26B85EC886CB523',
      'date': 'Fri, 24 Jul 2020 10:02:43 GMT',
      'content-type': 'application/xml'
    }
  },
  'Error': {
    'Message': 'The specified key does not exist.',
    'Code': 'NoSuchKey',
    'Key': 'thumbor/storage/8afd08cb72f146feb704b0506358937e/company_104259_logo_image_url.png'
  }
}

Versions

Old: 1.1 New: 6.2.15

Configuration

Configurations of old and new server are identical:

# Image Config
MAX_WIDTH = 2400
MAX_HEIGHT = 1600

# Uploading Config
UPLOAD_ENABLED = True
UPLOAD_MAX_SIZE = 20480000
UPLOAD_DELETE_ALLOWED = True

# S3 Config
LOADER = 'tc_aws.loaders.s3_loader'
STORAGE = 'tc_aws.storages.s3_storage'
RESULT_STORAGE = 'tc_aws.result_storages.s3_storage'
UPLOAD_PHOTO_STORAGE = 'tc_aws.storages.s3_storage'
TC_AWS_STORAGE_SSE = True

# Security Config
ALLOW_UNSAFE_URL = False

Any help would be greatly appreciated.

Bladrak commented 4 years ago

Hi, If I recall correctly (it's been quite a long time since), we had to change the upload paths at some point. The 6.x versions are not necessarily backwards compatible with the 1.x versions (that's one of the reasons why we changed the version scheme). Not sure how you may migrate your images in your buckets, it may be quite difficult.

Bladrak commented 1 year ago

Cleaning up old issues, feel free to reopen if it's still present with the latest releases

thumbor-community / aws