ImproperlyConfiguredException, could not load boto3 S3 bindings when running collectstatic in heroku

[jorgeas80]

I've managed to deploy my Django cookiecutter app in Heroku, but admin interface can't properly load CSS / JS resources because collectstatic fails.

This is the raw output when running collectstatic

ImportError: No module named 'boto3'
  File "storages/backends/s3boto3.py", line 17, in <module>
    import boto3.session

ImproperlyConfigured: Could not load Boto3's S3 bindings.
See https://github.com/boto/boto3
  File "manage.py", line 23, in <module>
    execute_from_command_line(sys.argv)
  File "django/core/management/__init__.py", line 367, in execute_from_command_line
    utility.execute()
  File "django/core/management/__init__.py", line 359, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "django/core/management/__init__.py", line 208, in fetch_command
    klass = load_command_class(app_name, subcommand)
  File "django/core/management/__init__.py", line 40, in load_command_class
    module = import_module('%s.management.commands.%s' % (app_name, name))
  File "importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 986, in _gcd_import
  File "<frozen importlib._bootstrap>", line 969, in _find_and_load
  File "<frozen importlib._bootstrap>", line 958, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 673, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 662, in exec_module
  File "<frozen importlib._bootstrap>", line 222, in _call_with_frames_removed
  File "collectfast/management/commands/collectstatic.py", line 7, in <module>
    from storages.backends.s3boto3 import S3Boto3Storage
  File "storages/backends/s3boto3.py", line 22, in <module>

I've manually installed boto3 in heroku by doing this:

heroku run pip install boto3

But nothing changes.

The problem is in file app/.heroku/python/lib/python3.5/site-packages/storages/backends/s3boto3.py. Specifically, those lines:

try:
    import boto3.session
    from boto3 import __version__ as boto3_version
    from botocore.client import Config
    from botocore.exceptions import ClientError
except ImportError:
    raise ImproperlyConfigured("Could not load Boto3's S3 bindings.\n"
                               "See https://github.com/boto/boto3")

To confirm, I enter the python shell

heroku run python manage.py shell

And try to manually import those files. For example

import boto3.session

Getting the same error

Traceback (most recent call last):
  File "/app/.heroku/python/lib/python3.5/code.py", line 91, in runcode
    exec(code, self.locals)
  File "<console>", line 1, in <module>
ImportError: No module named 'boto3'

Last test, run a pip freeze

heroku run pip freeze

amqp==1.4.9
anyjson==0.3.3
awesome-slugify==1.6.5
billiard==3.3.0.23
boto==2.45.0
celery==3.1.24
Collectfast==0.5.0
contextlib2==0.5.4
defusedxml==0.5.0
Django==1.10.5
django-allauth==0.30.0
django-anymail==0.7
django-appconf==1.0.2
django-braces==1.10.0
django-compressor==2.1
django-crispy-forms==1.6.1
django-dynamic-forms==0.5.3
django-environ==0.4.1
django-filer==1.2.6
django-fontawesome==0.3.1
django-location-field==2.0.2
django-model-utils==2.6
django-mptt==0.8.7
django-polymorphic==1.0.2
django-redis==4.7.0
django-storages==1.5.2 # This was manually installed. No changes. Still error.
django-storages-redux==1.3.2
easy-thumbnails==2.3
gevent==1.2.0
greenlet==0.4.12
gunicorn==19.6.0
kombu==3.0.37
oauthlib==2.0.1
olefile==0.44
Pillow==4.0.0
psycopg2==2.6.2
python3-openid==3.1.0
pytz==2016.10
PyYAML==3.12
raven==5.32.0
rcssmin==1.0.6
redis==2.10.5
regex==2017.2.8
requests==2.13.0
requests-oauthlib==0.8.0
rjsmin==1.0.12
six==1.10.0
Unidecode==0.4.20

No boto 3 listed, of course. No way of running collectstatic.

What am I doing wrong?

[jorgeas80]

More info: I've uninstalled boto 2.45 (or I thought so...), by pip uninstall boto

But still here

heroku run pip freeze | grep boto
boto==2.45.0

At same time, installing boto3 has no effect. And pip never says Requirement already satisfied. It always downloads and install boto3... but not really. Never shown in pip freeze.

And finally an even stranger... If I start a bash shell in my heroku instance

heroku run bash

Manually uninstall boto 2.45 and install boto3...

pip uninstall boto
pip install boto3

boto3 is correctly installed. But then I exit, and run a shell again.... and boto 3 dissapeared! boto 2.45 again here.

Why?? Am I entering into a "ghost" session and everything is deleted when I logout?

[jorgeas80]

Ok, solved. I had to replace

boto==2.45.0

With

boto3==1.4.4

In requirements/production.txt and push to heroku again.

I also had to replace

MEDIA_URL = 'https://s3.amazonaws.com/%s/media/' % AWS_STORAGE_BUCKET_NAME
STATIC_URL = 'https://s3.amazonaws.com/%s/static/' % AWS_STORAGE_BUCKET_NAME

with

MEDIA_URL = 'https://s3.amazonaws.com:443/%s/media/' % AWS_STORAGE_BUCKET_NAME
STATIC_URL = 'https://s3.amazonaws.com:443/%s/static/' % AWS_STORAGE_BUCKET_NAME

in config/settings/production.py. But not sure it this last thing is just my problem

I'm using python 3, by the way.

Not sure if this is pull request material or not. What do you think, guys?

[pydanny]

Which version of Python 3 are you using?

Anyway, a PR would be nice. I'll test it on Heroku and see what happens.

[jrthorne]

./manage.py collectstatic worked fine for static files for me. Media would not work with boto-rsync or django extensions. I fixed it by creating a bucket in the default US region, then used django extensions management command $ ./manage.py sync_s3 --media-only --s3host=s3.us-east-1.amazonaws.com

then copied and pasted to my EU central region bucket using the AWS website s3 console finder.

I used instructions here for setup https://wagtail.io/blog/amazon-s3-for-media-files/

[gurbanoglu]

Ok, solved. I had to replace

boto==2.45.0

With

boto3==1.4.4

In requirements/production.txt and push to heroku again.

I also had to replace

MEDIA_URL = 'https://s3.amazonaws.com/%s/media/' % AWS_STORAGE_BUCKET_NAME
STATIC_URL = 'https://s3.amazonaws.com/%s/static/' % AWS_STORAGE_BUCKET_NAME

with

MEDIA_URL = 'https://s3.amazonaws.com:443/%s/media/' % AWS_STORAGE_BUCKET_NAME
STATIC_URL = 'https://s3.amazonaws.com:443/%s/static/' % AWS_STORAGE_BUCKET_NAME

in config/settings/production.py. But not sure it this last thing is just my problem

I'm using python 3, by the way.

Not sure if this is pull request material or not. What do you think, guys?

I just wanted to say thank you! I changed "boto3=1.23.4" to "boto3=1.4.4" and now the django development server is running for me! Good work.