Encoding objects before hashing?

hvitis commented 4 years ago

Hello! Thank you for this great library. I am building a project with it and stumbled upon an Error I can not understand. Maybe someone did see anything similar?


lib/python3.6/site-packages/storages/backends/s3boto3.py:282: UserWarning: The default behavior of S3Boto3Storage is insecure and will change in django-storages 2.0. By default files and new buckets are saved with an ACL of 'public-read' (globally publicly readable). Version 2.0 will default to using the bucket's ACL. To opt into the new behavior set AWS_DEFAULT_ACL = None, otherwise to silence this warning explicitly set AWS_DEFAULT_ACL.
  "The default behavior of S3Boto3Storage is insecure and will change "
Traceback (most recent call last):
  File "manage.py", line 11, in <module>
    execute_from_command_line(sys.argv)
  File "lib/python3.6/site-packages/django/core/management/__init__.py", line 371, in execute_from_command_line
    utility.execute()
  File "lib/python3.6/site-packages/django/core/management/__init__.py", line 347, in execute
    django.setup()
  File "lib/python3.6/site-packages/django/__init__.py", line 24, in setup
    apps.populate(settings.INSTALLED_APPS)
  File "lib/python3.6/site-packages/django/apps/registry.py", line 112, in populate
    app_config.import_models()
  File "lib/python3.6/site-packages/django/apps/config.py", line 198, in import_models
    self.models_module = import_module(models_module_name)
  File "lib/python3.6/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 994, in _gcd_import
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 678, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "Courses-CMS/courses/models.py", line 17, in <module>
    from photologue.models import Gallery
  File "lib/python3.6/site-packages/photologue/models.py", line 74, in <module>
    "Signature: 8a477f597d28d172789f06886806bc55"))
  File "lib/python3.6/site-packages/django/core/files/storage.py", line 49, in save
    return self._save(name, content)
  File "lib/python3.6/site-packages/storages/backends/s3boto3.py", line 506, in _save
    self._save_content(obj, content, parameters=parameters)
  File "lib/python3.6/site-packages/storages/backends/s3boto3.py", line 521, in _save_content
    obj.upload_fileobj(content, ExtraArgs=put_parameters)
  File "lib/python3.6/site-packages/boto3/s3/inject.py", line 621, in object_upload_fileobj
    ExtraArgs=ExtraArgs, Callback=Callback, Config=Config)
  File "lib/python3.6/site-packages/boto3/s3/inject.py", line 539, in upload_fileobj
    return future.result()
  File "lib/python3.6/site-packages/s3transfer/futures.py", line 106, in result
    return self._coordinator.result()
  File "lib/python3.6/site-packages/s3transfer/futures.py", line 265, in result
    raise self._exception
  File "lib/python3.6/site-packages/s3transfer/tasks.py", line 126, in __call__
    return self._execute_main(kwargs)
  File "lib/python3.6/site-packages/s3transfer/tasks.py", line 150, in _execute_main
    return_value = self._main(**kwargs)
  File "lib/python3.6/site-packages/s3transfer/upload.py", line 692, in _main
    client.put_object(Bucket=bucket, Key=key, Body=body, **extra_args)
  File "lib/python3.6/site-packages/botocore/client.py", line 357, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "lib/python3.6/site-packages/botocore/client.py", line 642, in _make_api_call
    request_signer=self._request_signer, context=request_context)
  File "lib/python3.6/site-packages/botocore/hooks.py", line 360, in emit_until_response
    return self._emitter.emit_until_response(aliased_event_name, **kwargs)
  File "lib/python3.6/site-packages/botocore/hooks.py", line 243, in emit_until_response
    responses = self._emit(event_name, kwargs, stop_on_response=True)
  File "lib/python3.6/site-packages/botocore/hooks.py", line 211, in _emit
    response = handler(**kwargs)
  File "lib/python3.6/site-packages/botocore/handlers.py", line 212, in conditionally_calculate_md5
    calculate_md5(params, **kwargs)
  File "lib/python3.6/site-packages/botocore/handlers.py", line 190, in calculate_md5
    binary_md5 = _calculate_md5_from_file(body)
  File "lib/python3.6/site-packages/botocore/handlers.py", line 204, in _calculate_md5_from_file
    md5.update(chunk)
TypeError: Unicode-objects must be encoded before hashing

Looks like something connected with my Bucket because on local works fine. It makes me wonder because in the DOCS there is an example with using S3. I would be grateful for any clues.

hvitis commented 4 years ago

Short solution was editing models:

if not default_storage.exists(PHOTOLOGUE_CACHEDIRTAG):
    default_storage.save(PHOTOLOGUE_CACHEDIRTAG, ContentFile(
        b"Signature: 8a477f597d28d172789f06886806bc55"))

https://github.com/jschneier/django-storages/issues/708 https://github.com/druids/django-chamber/blob/master/chamber/storages/boto3.py#L44-L52

@richardbarran Thank you a lot for this repo. It's the best app for Galleries out there. You literally gave me days of life.

richardbarran commented 4 years ago

Hi, The compatibility with Boto/S3 was sent in as a PR a long time ago - I personally don't use it. From this ticket, and a few other recent tickets, it looks like there's an issue at the moment with using Photologue with S3. If someone wants to investigate and send in a PR I'll be happy to take a look at it.

hvitis commented 4 years ago

Hello @richardbarran

I created a pull to hotfix it https://github.com/richardbarran/django-photologue/pull/211

Thanks

richardbarran commented 4 years ago

Merge request has been merged - closing this ticket.

richardbarran / django-photologue

Encoding objects before hashing? #206