search

GrahamDumpleton / mod_wsgi

Source code for Apache/mod_wsgi.
Apache License 2.0
1.02k stars 268 forks source link

AWS 500 Internal Server Error error: Target WSGI script cannot be loaded as Python module #484

Closed django-dream closed 4 years ago

django-dream commented 4 years ago

Sir, I am trying to deploy a Django project to AWS EC2 but got the the error mentioned in the title. Could you please help? More info:

(1) I am using Ubuntu Server 18.04 LTS (HVM) from AWS EC2, python3 and Django 2.1.1. I am NOT using python virtual environment. I am trying to build a website (backed by EC2) to upload a video and store it to AWS S3 by using this example: https://github.com/sibtc/simple-s3-setup/tree/master/s3-example-static-and-media (which works well on my local machine).

(2) Error log (500 Internal Server Error): (A) Error 1: [Tue Nov 12 04:01:33.401351 2019] [wsgi:error] [pid 5979] [client 24.6.200.169:60887] mod_wsgi (pid=5979): Target WSGI script '/var/www/mysite/mysite/wsgi.py' cannot be loaded as Python module. [Tue Nov 12 04:01:33.401476 2019] [wsgi:error] [pid 5979] [client 24.6.200.169:60887] mod_wsgi (pid=5979): Exception occurred processing WSGI script '/var/www/mysite/mysite/wsgi.py'. [Tue Nov 12 04:01:33.407189 2019] [wsgi:error] [pid 5979] [client 24.6.200.169:60887] Traceback (most recent call last): [Tue Nov 12 04:01:33.407241 2019] [wsgi:error] [pid 5979] [client 24.6.200.169:60887] File "/var/www/mysite/mysite/wsgi.py", line 16, in [Tue Nov 12 04:01:33.407247 2019] [wsgi:error] [pid 5979] [client 24.6.200.169:60887] application = get_wsgi_application() [Tue Nov 12 04:01:33.407255 2019] [wsgi:error] [pid 5979] [client 24.6.200.169:60887] File "/usr/local/lib/python3.6/dist-packages/django/core/wsgi.py", line 12, in get_wsgi_application [Tue Nov 12 04:01:33.407259 2019] [wsgi:error] [pid 5979] [client 24.6.200.169:60887] django.setup(set_prefix=False) [Tue Nov 12 04:01:33.407277 2019] [wsgi:error] [pid 5979] [client 24.6.200.169:60887] File "/usr/local/lib/python3.6/dist-packages/django/init.py", line 24, in setup [Tue Nov 12 04:01:33.407281 2019] [wsgi:error] [pid 5979] [client 24.6.200.169:60887] apps.populate(settings.INSTALLED_APPS) [Tue Nov 12 04:01:33.407287 2019] [wsgi:error] [pid 5979] [client 24.6.200.169:60887] File "/usr/local/lib/python3.6/dist-packages/django/apps/registry.py", line 91, in populate [Tue Nov 12 04:01:33.407291 2019] [wsgi:error] [pid 5979] [client 24.6.200.169:60887] app_config = AppConfig.create(entry) [Tue Nov 12 04:01:33.407297 2019] [wsgi:error] [pid 5979] [client 24.6.200.169:60887] File "/usr/local/lib/python3.6/dist-packages/django/apps/config.py", line 90, in create [Tue Nov 12 04:01:33.407301 2019] [wsgi:error] [pid 5979] [client 24.6.200.169:60887] module = import_module(entry) [Tue Nov 12 04:01:33.407306 2019] [wsgi:error] [pid 5979] [client 24.6.200.169:60887] File "/usr/lib/python3.6/importlib/init.py", line 126, in import_module [Tue Nov 12 04:01:33.407311 2019] [wsgi:error] [pid 5979] [client 24.6.200.169:60887] return _bootstrap._gcd_import(name[level:], package, level) [Tue Nov 12 04:01:33.407316 2019] [wsgi:error] [pid 5979] [client 24.6.200.169:60887] File "", line 994, in _gcd_import [Tue Nov 12 04:01:33.407323 2019] [wsgi:error] [pid 5979] [client 24.6.200.169:60887] File "", line 971, in _find_and_load [Tue Nov 12 04:01:33.407329 2019] [wsgi:error] [pid 5979] [client 24.6.200.169:60887] File "", line 953, in _find_and_load_unlocked [Tue Nov 12 04:01:33.407349 2019] [wsgi:error] [pid 5979] [client 24.6.200.169:60887] ModuleNotFoundError: No module named 'storages'

(B) Error 2: BUT, sometimes the error is like this: [Tue Nov 12 04:28:07.880332 2019] [wsgi:error] [pid 5977] [client 24.6.200.169:61304] mod_wsgi (pid=5977): Target WSGI script '/var/www/mysite/mysite/wsgi.py' cannot be loaded as Python module. [Tue Nov 12 04:28:07.880406 2019] [wsgi:error] [pid 5977] [client 24.6.200.169:61304] mod_wsgi (pid=5977): Exception occurred processing WSGI script '/var/www/mysite/mysite/wsgi.py'. [Tue Nov 12 04:28:07.880533 2019] [wsgi:error] [pid 5977] [client 24.6.200.169:61304] Traceback (most recent call last): [Tue Nov 12 04:28:07.880570 2019] [wsgi:error] [pid 5977] [client 24.6.200.169:61304] File "/var/www/mysite/mysite/wsgi.py", line 16, in [Tue Nov 12 04:28:07.880576 2019] [wsgi:error] [pid 5977] [client 24.6.200.169:61304] application = get_wsgi_application() [Tue Nov 12 04:28:07.880584 2019] [wsgi:error] [pid 5977] [client 24.6.200.169:61304] File "/usr/local/lib/python3.6/dist-packages/django/core/wsgi.py", line 12, in get_wsgi_application [Tue Nov 12 04:28:07.880589 2019] [wsgi:error] [pid 5977] [client 24.6.200.169:61304] django.setup(set_prefix=False) [Tue Nov 12 04:28:07.880596 2019] [wsgi:error] [pid 5977] [client 24.6.200.169:61304] File "/usr/local/lib/python3.6/dist-packages/django/init.py", line 24, in setup [Tue Nov 12 04:28:07.880600 2019] [wsgi:error] [pid 5977] [client 24.6.200.169:61304] apps.populate(settings.INSTALLED_APPS) [Tue Nov 12 04:28:07.880606 2019] [wsgi:error] [pid 5977] [client 24.6.200.169:61304] File "/usr/local/lib/python3.6/dist-packages/django/apps/registry.py", line 83, in populate [Tue Nov 12 04:28:07.880610 2019] [wsgi:error] [pid 5977] [client 24.6.200.169:61304] raise RuntimeError("populate() isn't reentrant") [Tue Nov 12 04:28:07.880626 2019] [wsgi:error] [pid 5977] [client 24.6.200.169:61304] RuntimeError: populate() isn't reentrant

(Please refer to https://github.com/sibtc/simple-s3-setup/blob/master/s3-example-static-and-media/mysite for the (3) and (5), where my only difference from it is the AWS setting in settings.py) (3) settings.py: ... INSTALLED_APPS = [ ..., 'storages', 'mysite.core', ] ... STATICFILES_STORAGE = 'storages.backends.s3boto3.S3Boto3Storage' STATIC_URL = "https://%s/%s/" % (AWS_S3_CUSTOM_DOMAIN, AWS_LOCATION) DEFAULT_FILE_STORAGE = 'mysite.storage_backends.MediaStorage'

(4) /etc/apache2/apache2.conf: WSGIScriptAlias / /var/www/mysite/mysite/wsgi.py WSGIPythonPath /var/www/mysite <Directory /var/www/mysite/mysite>

Require all granted

(5) wsgi.py: import os from django.core.wsgi import get_wsgi_application os.environ.setdefault("DJANGO_SETTINGS_MODULE", "mysite.settings") application = get_wsgi_application()

(6) I have followed your instruction to install mod_wsgi: https://github.com/GrahamDumpleton/mod_wsgi/issues/378#issuecomment-446473948 (7) I have given the wsgi.py file the permission (from one of your suggestions): sudo chmod 0777 mysite/wsgi.py

Thank you very much.

GrahamDumpleton commented 4 years ago

I can't see that you have told mod_wsgi about any Python virtual environment you are using. Are you using one? Where is the storages module installed to?

See:

django-dream commented 4 years ago

I am not using python virtual environment. Thanks for pointing out the problem with storage module. I am fixing it and also trying to conquer the wsgi problem with several new ideas. I will keep updating the post with any progresses. Thanks.

django-dream commented 4 years ago

@GrahamDumpleton , I have updated the problem statement. I tried to debug by playing with "storages" and wsgi, but sometimes I got the error related to storages and sometimes related to "populate() isn't reentrant", but both of them have the message of "Target WSGI script cannot be loaded as Python module". Can you please provide any advice? Thanks a lot

GrahamDumpleton commented 4 years ago

The message about "populate() isn't reentrant" is a secondary error which can be ignored. You need to work out what the original error for that process was.

Are you saying that the "storages" module cannot be found still?

Where is the "storages" module installed?

What do you get if from command line Python interpreter on target host you execute:

import sys
print(sys.prefix)
import storages
django-dream commented 4 years ago

The error shows "ModuleNotFoundError: No module named 'storages'". However, in both the /home/ubuntu directory or the project directory (/var/www/mysite/mysite), I can "import storages", where I got "/usr" once typing "print(sys.prefix)" and got no error once typing "import storages". I installed "storages" module in /home/ubuntu by using "pip3 install django-storages" (and "pip3 install boto3" for boto3) by following https://simpleisbetterthancomplex.com/tutorial/2017/08/01/how-to-setup-amazon-s3-in-a-django-project.html.

GrahamDumpleton commented 4 years ago

How did you install the Python packages since you aren't using a Python virtual environment?

Did you use:

pip install --user ...

Because Apache isn't running as you, but a special Apache user, it will not read packages from a per user site packages directory. This is why you need to use a Python virtual environment.

GrahamDumpleton commented 4 years ago

Also, what do you get from interpreter when you run:

import storages
print(storages.__file__)

Forgot to include the last line last time when asked to test import.

django-dream commented 4 years ago

No problem. Thanks. I got "/home/ubuntu/.local/lib/python3.6/site-packages/storages/init.py" once typing print(storages.file).

GrahamDumpleton commented 4 years ago

Which means you must have used --user.

Although it may work by using:

WSGIPythonPath /var/www/mysite:/home/ubuntu/.local/lib/python3.6/site-packages

you are better off using a Python virtual environment, installing packages into it and then configure mod_wsgi to use the Python virtual environment.

django-dream commented 4 years ago

I am now trying to make it work before I try virtual environment.

I got the same error when I use "WSGIPythonPath /var/www/mysite:/home/ubuntu/.local/lib/python3.6/site-packages" and several variations.

Here is "/etc/apache2/apache2.conf" (the content after "Require all granted " is always truncated): WSGIScriptAlias / /var/www/mysite/mysite/wsgi.py WSGIPythonPath /var/www/mysite:/home/ubuntu/.local/lib/python3.6/site-packages <Directory /var/www/mysite/mysite>

Require all granted

This is the error log: [Wed Nov 13 05:04:59.724336 2019] [wsgi:error] [pid 10224] [client 24.6.200.169:50873] mod_wsgi (pid=10224): Target WSGI script '/var/www/mysite/mysite/wsgi.py' cannot be loaded as Python module. [Wed Nov 13 05:04:59.724440 2019] [wsgi:error] [pid 10224] [client 24.6.200.169:50873] mod_wsgi (pid=10224): Exception occurred processing WSGI script '/var/www/mysite/mysite/wsgi.py'. [Wed Nov 13 05:04:59.725313 2019] [wsgi:error] [pid 10224] [client 24.6.200.169:50873] Traceback (most recent call last): [Wed Nov 13 05:04:59.725362 2019] [wsgi:error] [pid 10224] [client 24.6.200.169:50873] File "/var/www/mysite/mysite/wsgi.py", line 16, in [Wed Nov 13 05:04:59.725369 2019] [wsgi:error] [pid 10224] [client 24.6.200.169:50873] application = get_wsgi_application() [Wed Nov 13 05:04:59.725376 2019] [wsgi:error] [pid 10224] [client 24.6.200.169:50873] File "/usr/local/lib/python3.6/dist-packages/django/core/wsgi.py", line 12, in get_wsgi_application [Wed Nov 13 05:04:59.725381 2019] [wsgi:error] [pid 10224] [client 24.6.200.169:50873] django.setup(set_prefix=False) [Wed Nov 13 05:04:59.725387 2019] [wsgi:error] [pid 10224] [client 24.6.200.169:50873] File "/usr/local/lib/python3.6/dist-packages/django/init.py", line 24, in setup [Wed Nov 13 05:04:59.725391 2019] [wsgi:error] [pid 10224] [client 24.6.200.169:50873] apps.populate(settings.INSTALLED_APPS) [Wed Nov 13 05:04:59.725397 2019] [wsgi:error] [pid 10224] [client 24.6.200.169:50873] File "/usr/local/lib/python3.6/dist-packages/django/apps/registry.py", line 91, in populate [Wed Nov 13 05:04:59.725401 2019] [wsgi:error] [pid 10224] [client 24.6.200.169:50873] app_config = AppConfig.create(entry) [Wed Nov 13 05:04:59.725407 2019] [wsgi:error] [pid 10224] [client 24.6.200.169:50873] File "/usr/local/lib/python3.6/dist-packages/django/apps/config.py", line 90, in create [Wed Nov 13 05:04:59.725411 2019] [wsgi:error] [pid 10224] [client 24.6.200.169:50873] module = import_module(entry) [Wed Nov 13 05:04:59.725417 2019] [wsgi:error] [pid 10224] [client 24.6.200.169:50873] File "/usr/lib/python3.6/importlib/init.py", line 126, in import_module [Wed Nov 13 05:04:59.725421 2019] [wsgi:error] [pid 10224] [client 24.6.200.169:50873] return _bootstrap._gcd_import(name[level:], package, level) [Wed Nov 13 05:04:59.725441 2019] [wsgi:error] [pid 10224] [client 24.6.200.169:50873] File "", line 994, in _gcd_import [Wed Nov 13 05:04:59.725448 2019] [wsgi:error] [pid 10224] [client 24.6.200.169:50873] File "", line 971, in _find_and_load [Wed Nov 13 05:04:59.725453 2019] [wsgi:error] [pid 10224] [client 24.6.200.169:50873] File "", line 953, in _find_and_load_unlocked [Wed Nov 13 05:04:59.725473 2019] [wsgi:error] [pid 10224] [client 24.6.200.169:50873] ModuleNotFoundError: No module named 'storages'

Thanks a lot.

GrahamDumpleton commented 4 years ago

Likely the /home/ubuntu directory is not accessible to the Apache user. What do you get when you run:

ls -lasd /home/ubuntu
django-dream commented 4 years ago

I got "4 drwxrwxr-x 9 ubuntu ubuntu 4096 Nov 13 05:05 /home/ubuntu"

GrahamDumpleton commented 4 years ago

Check permissions on all directories down to /home/ubuntu/.local/lib/python3.6/site-packages and make sure some sub directory has more restrictive permissions. Also check file permissions in the package directory.

django-dream commented 4 years ago

Thanks a lot. It works once I used "chmod -R 777 /home/ubuntu/my_project" to issue the access permission to everyone (a bad operation. Just want to make it work :) ).

I got the error of "attempt to write a readonly database" in "/home/ubuntu/.local/lib/python3.6/site-packages/django/db/backends/sqlite3/base.py in execute, line 383", when I upload a video. HOWEVER, it is not related to the "storages" module problem here. I am working on it and will also try virtual environment. Thanks, again!

GrahamDumpleton commented 4 years ago

Make sure read: