S3 Credentials Issue after upgrading to 1.14.4

[trumpet2012]

We have a lambda function that gets called as part of our cloudformation deployment to run collectstatic. After upgrading django-storages from 1.14.3 to 1.14.4 we started seeing the following errors when running collectstatic:

TypeError: sequence item 0: expected str instance, NoneType found
  File "crhelper/resource_helper.py", line 204, in _wrap_function
    self.PhysicalResourceId = func(self._event, self._context) if func else ''
  File "handlers/manage_custom_resource.py", line 11, in run_manage_command
    manage_handler(event["ResourceProperties"], context)
  File "handlers/manage.py", line 25, in handler
    call_command(*command)
  File "django/core/management/__init__.py", line 194, in call_command
    return command.execute(*args, **defaults)
  File "django/core/management/base.py", line 458, in execute
    output = self.handle(*args, **options)
  File "django/contrib/staticfiles/management/commands/collectstatic.py", line 209, in handle
    collected = self.collect()
  File "django/contrib/staticfiles/management/commands/collectstatic.py", line 135, in collect
    handler(path, prefixed_path, storage)
  File "django/contrib/staticfiles/management/commands/collectstatic.py", line 378, in copy_file
    self.storage.save(prefixed_path, source_file)
  File "django/core/files/storage/base.py", line 49, in save
    name = self._save(name, content)
  File "storages/backends/s3.py", line 564, in _save
    obj.upload_fileobj(content, ExtraArgs=params, Config=self.transfer_config)
  File "boto3/s3/inject.py", line 731, in object_upload_fileobj
    return self.meta.client.upload_fileobj(
  File "boto3/s3/inject.py", line 642, in upload_fileobj
    return future.result()
  File "s3transfer/futures.py", line 103, in result
    return self._coordinator.result()
  File "s3transfer/futures.py", line 266, in result
    raise self._exception
  File "s3transfer/tasks.py", line 139, in __call__
    return self._execute_main(kwargs)
  File "s3transfer/tasks.py", line 162, in _execute_main
    return_value = self._main(**kwargs)
  File "s3transfer/upload.py", line 764, in _main
    client.put_object(Bucket=bucket, Key=key, Body=body, **extra_args)
  File "botocore/client.py", line 565, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "botocore/client.py", line 1001, in _make_api_call
    http, parsed_response = self._make_request(
  File "botocore/client.py", line 1027, in _make_request
    return self._endpoint.make_request(operation_model, request_dict)
  File "botocore/endpoint.py", line 119, in make_request
    return self._send_request(request_dict, operation_model)
  File "botocore/endpoint.py", line 198, in _send_request
    request = self.create_request(request_dict, operation_model)
  File "botocore/endpoint.py", line 134, in create_request
    self._event_emitter.emit(
  File "botocore/hooks.py", line 412, in emit
    return self._emitter.emit(aliased_event_name, **kwargs)
  File "botocore/hooks.py", line 256, in emit
    return self._emit(event_name, kwargs)
  File "botocore/hooks.py", line 239, in _emit
    response = handler(**kwargs)
  File "botocore/signers.py", line 105, in handler
    return self.sign(operation_name, request)
  File "botocore/signers.py", line 199, in sign
    auth.add_auth(request)
  File "botocore/auth.py", line 432, in add_auth
    self._inject_signature_to_request(request, signature)
  File "botocore/auth.py", line 435, in _inject_signature_to_request
    auth_str = ['AWS4-HMAC-SHA256 Credential=%s' % self.scope(request)]
  File "botocore/auth.py", line 384, in scope
    return '/'.join(scope)

Looking at the error in sentry we are able to see that the None in the list of scopes is coming from self.credentials.access_key.

Reverting back to 1.14.3 fixes the issue.

Other dep versions

django==4.2.14
botocore==1.34.146
boto3==1.34.146
s3transfer==0.10.2

[jschneier]

Thanks for the report. I'm not sure how to reproduce — can you shed any light on your debugging? It seems like to me that https://github.com/jschneier/django-storages/commit/9fca46c76f9b10131f3329a013cd2a67e7fdd4f5 is the culprit, could you try reverting just that commit and seeing if that fixes it?

Touching anything with S3 auth is always a nightmare, too many ways to slice it.

[trumpet2012]

I'm working on getting a reproducible CDK stack and django app that I can share to recreate the issue. We have to do some hacky things in CDK to get cloudformation to run our lambda management commands as part of the deployment. I suspect it's going to be something with how that is implemented.