Pomax / filebrowser_s3

an AWS S3 fix for Mezzanine's media manager
Mozilla Public License 2.0
9 stars 3 forks source link

error regarding encoding #4

Open Jamie-BitFlight opened 7 years ago

Jamie-BitFlight commented 7 years ago
"storage": {"stacktrace": "Traceback (most recent call last):
  File \"/usr/local/lib/python3.5/site-packages/watchman/decorators.py\", line 17, in wrapped
    response = func(*args, **kwargs)
  File \"/usr/local/lib/python3.5/site-packages/watchman/checks.py\", line 63, in _check_storage
    path = default_storage.save(filename, ContentFile(content))
  File \"/usr/local/lib/python3.5/site-packages/django/core/files/storage.py\", line 54, in save
    return self._save(name, content)
  File \"/usr/local/lib/python3.5/site-packages/storages/backends/s3boto3.py\", line 452, in _save
    self._save_content(obj, content, parameters=parameters)
  File \"/usr/local/lib/python3.5/site-packages/storages/backends/s3boto3.py\", line 467, in _save_content
    obj.upload_fileobj(content, ExtraArgs=put_parameters)
  File \"/usr/local/lib/python3.5/site-packages/boto3/s3/inject.py\", line 509, in object_upload_fileobj
    ExtraArgs=ExtraArgs, Callback=Callback, Config=Config)
  File \"/usr/local/lib/python3.5/site-packages/boto3/s3/inject.py\", line 427, in upload_fileobj
    return future.result()
  File \"/usr/local/lib/python3.5/site-packages/s3transfer/futures.py\", line 73, in result
    return self._coordinator.result()
  File \"/usr/local/lib/python3.5/site-packages/s3transfer/futures.py\", line 233, in result
    raise self._exception
  File \"/usr/local/lib/python3.5/site-packages/s3transfer/tasks.py\", line 126, in __call__
    return self._execute_main(kwargs)
  File \"/usr/local/lib/python3.5/site-packages/s3transfer/tasks.py\", line 150, in _execute_main
    return_value = self._main(**kwargs)
  File \"/usr/local/lib/python3.5/site-packages/s3transfer/upload.py\", line 679, in _main
    client.put_object(Bucket=bucket, Key=key, Body=body, **extra_args)
  File \"/usr/local/lib/python3.5/site-packages/botocore/client.py\", line 253, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File \"/usr/local/lib/python3.5/site-packages/botocore/client.py\", line 525, in _make_api_call
    request_signer=self._request_signer, context=request_context)
  File \"/usr/local/lib/python3.5/site-packages/botocore/hooks.py\", line 242, in emit_until_response
    responses = self._emit(event_name, kwargs, stop_on_response=True)
  File \"/usr/local/lib/python3.5/site-packages/botocore/hooks.py\", line 210, in _emit
    response = handler(**kwargs)
  File \"/usr/local/lib/python3.5/site-packages/botocore/handlers.py\", line 200, in conditionally_calculate_md5
    calculate_md5(params, **kwargs)
  File \"/usr/local/lib/python3.5/site-packages/botocore/handlers.py\", line 178, in calculate_md5
    binary_md5 = _calculate_md5_from_file(body)
  File \"/usr/local/lib/python3.5/site-packages/botocore/handlers.py\", line 192, in _calculate_md5_from_file
    md5.update(chunk)
TypeError: Unicode-objects must be encoded before hashing
", "ok": false, "error": "Unicode-objects must be encoded before hashing"}
Pomax commented 7 years ago

hm, python 3.5 should be modern enough, and the code we rely on should already work without issues, thanks for the bug report, we'll look into this. Can you explain the setup you're using, or if possible, do you have STR/minimal reproducible, verifiable example code that we can run?

(we can't seem to reproduce that error here using python 3.6 running inside a virtual environment)

Jamie-BitFlight commented 7 years ago

I have it in a docker dockerfile FROM python:3.5 ENV PYTHONUNBUFFERED 1 RUN mkdir /code WORKDIR /code ADD requirements.txt /code/ RUN pip install -r requirements.txt ADD . /code/`

requirements.txt cat requirements.txt -e git://github.com/boto/botocore.git@develop#egg=botocore -e git://github.com/boto/jmespath.git@develop#egg=jmespath -e git://github.com/boto/s3transfer.git@develop#egg=s3transfer -e git://github.com/boto/boto3.git@develop#egg=boto3 -e 'git+https://github.com/jschneier/django-storages.git#egg=django-storages' nose mock wheel simplejson python-json-logger json-logging-py setuptools pillow mezzanine psycopg2 python-dateutil filebrowser_s3 django-watchman paste gevent django-redis gunicorn

and a snippit from my settings.py (i replaced the aws id and secret with made up ones) `# These will be added to INSTALLED_APPS, only if available. OPTIONAL_APPS = ( "filebrowser_s3", "theme", "storages", "watchman",

"mezzanine.accounts",

# "mezzanine.mobile",
#"django_singleton_admin",
"theme.blog_mods",
"theme.portfolio",
"debug_toolbar",
"django_extensions",
"compressor",
"gunicorn",
PACKAGE_NAME_FILEBROWSER,
PACKAGE_NAME_GRAPPELLI,

) AWS_S3_REGION_NAME = 'eu-west-2' AWS_S3_CUSTOM_DOMAIN = 'assets.example.com' AWS_STORAGE_BUCKET_NAME = 'www.example.com' #bucket name is www. instead of assets. because i am lazy AWS_ACCESS_KEY_ID = 'ALIAJ6XK5H7OF2438THQ' AWS_SECRET_ACCESS_KEY = '0UqSjhR82vdjcJD6jcbtee1A4zjOb/ycSe8vO40+'

use filebrowser_s3 rather than plain filebrowser_safe

DEFAULT_FILE_STORAGE = 'filebrowser_s3.storage.S3MediaStorage'

AWS_PRELOAD_METADATA = True

AWS_LOCATION = env('AWS_STORAGE_ROOT', default=None)

AWS_LOCATION = 'media' MEDIA_URL = 'https://' + AWS_S3_CUSTOM_DOMAIN + '/' + AWS_LOCATION + '/' MEDIA_ROOT = '' STATICFILES_LOCATION = 'static' STATICFILES_STORAGE = 'custom_storages.StaticStorage' STATIC_URL = "https://%s/%s/" % (AWS_S3_CUSTOM_DOMAIN, STATICFILES_LOCATION)

Make sure to explicitly set this to an empty string

FILEBROWSER_DIRECTORY = ''

##################

LOCAL SETTINGS

################## SECURE_PROXY_SSL_HEADER = ('HTTP_X_FORWARDED_PROTO', 'https')`