UnicodeEncodeError: Upload to S3 bucket fails

[cybertschunk]

hi guys,

I have problems getting the S3-Backend to work. The deployment to a elastic-beanstalk-application fails with this error message:

Traceback (most recent call last):
  File "manage.py", line 10, in <module>
  execute_from_command_line(sys.argv)
  File "/opt/python/run/venv/local/lib/python3.6/site-packages/django/core/management/__init__.py", line 381, in execute_from_command_line
  utility.execute()
  File "/opt/python/run/venv/local/lib/python3.6/site-packages/django/core/management/__init__.py", line 375, in execute
  self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/opt/python/run/venv/local/lib/python3.6/site-packages/django/core/management/base.py", line 323, in run_from_argv
  self.execute(*args, **cmd_options)
  File "/opt/python/run/venv/local/lib/python3.6/site-packages/django/core/management/base.py", line 364, in execute
  output = self.handle(*args, **options)
  File "/opt/python/run/venv/local/lib/python3.6/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 188, in handle
  collected = self.collect()
  File "/opt/python/run/venv/local/lib/python3.6/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 114, in collect
  handler(path, prefixed_path, storage)
  File "/opt/python/run/venv/local/lib/python3.6/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 342, in copy_file
  if not self.delete_file(path, prefixed_path, source_storage):
  File "/opt/python/run/venv/local/lib/python3.6/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 249, in delete_file
  if self.storage.exists(prefixed_path):
  File "/opt/python/run/venv/local/lib/python3.6/site-packages/storages/backends/s3boto3.py", line 520, in exists
  self.connection.meta.client.head_object(Bucket=self.bucket_name, Key=name)
  File "/opt/python/run/venv/local/lib/python3.6/site-packages/botocore/client.py", line 357, in _api_call
  return self._make_api_call(operation_name, kwargs)
  File "/opt/python/run/venv/local/lib/python3.6/site-packages/botocore/client.py", line 634, in _make_api_call
  api_params, operation_model, context=request_context)
  File "/opt/python/run/venv/local/lib/python3.6/site-packages/botocore/client.py", line 682, in _convert_to_request_dict
  api_params, operation_model)
  File "/opt/python/run/venv/local/lib/python3.6/site-packages/botocore/validate.py", line 299, in serialize_to_request
  operation_model)
  File "/opt/python/run/venv/local/lib/python3.6/site-packages/botocore/serialize.py", line 449, in serialize_to_request
  partitioned['uri_path_kwargs'])
  File "/opt/python/run/venv/local/lib/python3.6/site-packages/botocore/serialize.py", line 476, in _render_uri_template
  params[template_param[:-1]], safe='/~')
  File "/opt/python/run/venv/local/lib/python3.6/site-packages/botocore/utils.py", line 540, in percent_encode
  input_str = input_str.encode('utf-8')
  UnicodeEncodeError: 'utf-8' codec can't encode characters in position 24-25: surrogates not allowed
   (ElasticBeanstalk::ExternalInvocationError)

Here are my django-settings:

AWS_ACCESS_KEY_ID = os.environ.get('S3_ACCESS_KEY_ID')
AWS_SECRET_ACCESS_KEY = os.environ.get('S3_SECRET_ACCESS_KEY')

# Configure which endpoint to send files to, and retrieve files from.
AWS_STORAGE_BUCKET_NAME = '[…]'
AWS_S3_REGION_NAME = 'eu-central-1'
AWS_S3_ENDPOINT_URL = "[…].s3.eu-central-1.amazonaws.com"
AWS_S3_CUSTOM_DOMAIN = "https://cdn.[…]"
AWS_LOCATION = 'files'
AWS_DEFAULT_ACL = 'public-read'

# General optimization for faster delivery
AWS_IS_GZIPPED = True
AWS_S3_OBJECT_PARAMETERS = {
    'CacheControl': 'max-age=86400',
}

STATIC_ROOT = 'static'
MEDIA_ROOT = 'media'
STATIC_URL = f"https://{AWS_S3_ENDPOINT_URL}/{STATIC_ROOT}/"
MEDIA_URL = f"https://{AWS_S3_ENDPOINT_URL}/{MEDIA_ROOT}/"

These steps reproduce the error:

execute: source /opt/python/run/venv/bin/activate && python manage.py collectstatic --noinput

I guess, it's something host related, since the command executes perfectly on my local machine.

Do you have any ideas what could cause the error? That would be awesome! I am unfortunately really stuck here…

[aravindmrao]

@cybertschunk I have a similar problem and am stuck too. Did you resolve this issue by any chance?

[cybertschunk]

hey @aravindmrao, yeah I managed to resolve this issue by setting the environment variable LC_ALL to my national locale (de_DE.UTF-8) on the server, that performs the task.Since I use elastic beanstalk for the project, I could easily do this in the software properties of the project configuration

[aravindmrao]

thank you @cybertschunk! mine was a similar issue but not exactly the same and it came down to setting a parameter in settings.py as well. Your reply helped me narrow it down to the settings.py and finally fix it. Thank you once again!!

[jakobkarlstrand]

thank you @cybertschunk! mine was a similar issue but not exactly the same and it came down to setting a parameter in settings.py as well. Your reply helped me narrow it down to the settings.py and finally fix it. Thank you once again!!

How did you solve this? I have the same error