search

nginx / unit

NGINX Unit - universal web app server - a lightweight and versatile open source server that simplifies the application stack by natively executing application code across eight different programming language runtimes.
https://unit.nginx.org
Apache License 2.0
5.35k stars 321 forks source link

ModuleNotFoundError: No module named 'encodings' #516

Open deepakkumar-19490 opened 3 years ago

deepakkumar-19490 commented 3 years ago

I keep getting: ModuleNotFoundError: No module named 'encodings' in the unit error logs when trying to add new configuration.

Fatal Python error: Py_Initialize: Unable to get the locale encoding ModuleNotFoundError: No module named 'encodings'

2020/12/31 19:51:50 [notice] 13294#13294 process 13297 exited with code 0
2020/12/31 19:51:50 [info] 15500#15500 discovery started
2020/12/31 19:51:50 [notice] 15500#15500 module: java 11.0.8 "/usr/lib/unit/modules/java11.unit.so"
2020/12/31 19:51:50 [notice] 15500#15500 module: perl 5.26.1 "/usr/lib/unit/modules/perl.unit.so"
2020/12/31 19:51:50 [notice] 15500#15500 module: php 7.2.24-0ubuntu0.18.04.6 "/usr/lib/unit/modules/php.unit.so"
2020/12/31 19:51:50 [notice] 15500#15500 module: python 2.7.17 "/usr/lib/unit/modules/python2.7.unit.so"
2020/12/31 19:51:50 [notice] 15500#15500 module: python 3.6.9 "/usr/lib/unit/modules/python3.6.unit.so"
2020/12/31 19:51:50 [notice] 15500#15500 module: python 3.7.5 "/usr/lib/unit/modules/python3.7.unit.so"
2020/12/31 19:51:50 [notice] 15500#15500 module: ruby 2.5.1 "/usr/lib/unit/modules/ruby.unit.so"
2020/12/31 19:51:50 [info] 15499#15499 controller started
2020/12/31 19:51:50 [notice] 15499#15499 process 15500 exited with code 0
2020/12/31 19:51:50 [info] 15514#15514 router started
2020/12/31 19:51:50 [info] 15514#15514 OpenSSL 1.1.1  11 Sep 2018, 1010100f
2020/12/31 19:52:40 [info] 15597#15597 "example_python" application started
2020/12/31 19:52:40 [info] 15602#15602 "example_python" application started
2020/12/31 19:54:20 [info] 15766#15766 "example_python" application started
2020/12/31 19:54:20 [info] 15767#15767 "example_python" application started
2020/12/31 19:54:20 [notice] 15499#15499 process 15597 exited with code 0
2020/12/31 19:54:20 [notice] 15499#15499 process 15602 exited with code 0
2020/12/31 19:57:49 [info] 16117#16117 "django" application started
Fatal Python error: Py_Initialize: Unable to get the locale encoding
ModuleNotFoundError: No module named 'encodings'

Current thread 0x00007f64d9a9ad80 (most recent call first):
2020/12/31 19:57:49 [alert] 15499#15499 process 16117 exited on signal 6 (core dumped)
2020/12/31 19:57:49 [warn] 15514#15514 failed to start application "django"
2020/12/31 19:57:49 [alert] 15514#15514 failed to apply new conf

sudo curl -X PUT --data-binary @unit.config --unix-socket /var/run/control.unit.sock http://localhost/config unit.config

 "applications": {
        "django": {
            "type": "python 3.7",
            "user": "deepak",
            "group": "deepak",
            "path": "/home/deepak/PycharmProjects/project/venv",
            "home": "/home/deepak/PycharmProjects/project",
            "module": "wsgi",
            "environment": {
                "DJANGO_SETTINGS_MODULE": "project.settings",
                "DB_ENGINE": "django.db.backends.postgresql",
                "DB_NAME": "project",
                "DB_HOST": "127.0.0.1",
                "DB_PORT": "5434"
            }
        }
    },

    "listeners": {
        "*:8400": {
            "pass": "applications/django"
        }
    }
}
deepakkumar-19490 commented 3 years ago

Issue is present in python 3.7, not in python 3.6

mar0x commented 3 years ago

Hello, Please check your venv has modules for python3.7. Most likely it was created for python3.6:

$ ls venv/lib/
python3.6
RiderALT commented 2 years ago

I also hav the same issue and I do not understand how to run unit with django. Python 3.9. unit 1.25.0

$ source venv/bin/activate
(venv) $ pip list --local
Package         Version
--------------- ---------
asgiref         3.4.1
certifi         2021.10.8
chardet         4.0.0
Django          3.2.5
django-ckeditor 6.1.0
django-js-asset 1.2.2
gunicorn        20.1.0
idna            2.10
pip             21.3.1
python-dateutil 2.8.2
pytz            2021.1
requests        2.25.1
setuptools      56.0.0
six             1.16.0
sqlparse        0.4.2
urllib3         1.26.6
(venv)

config:

    "applications": {
        "frontend": {
            "type": "python 3.9",
            "path": "/home/frontend/altrepo_front/",
            "home": "/home/frontend/altrepo_front/venv",
            "module": "altrepo_front.wsgi",
            "environment": {
                "DJANGO_SETTINGS_MODULE": "altrepo_front.settings",
                "DB_ENGINE": "django.db.backends.sqlite3"
            }
        }
    }

log:

2021/10/27 10:10:40 [info] 10536#10536 "frontend" application started
Python path configuration:
  PYTHONHOME = '/home/frontend/altrepo_front/venv'
  PYTHONPATH = (not set)
  program name = 'python3'
  isolated = 0
  environment = 1
  user site = 1
  import site = 1
  sys._base_executable = '/usr/bin/python3'
  sys.base_prefix = '/home/frontend/altrepo_front/venv'
  sys.base_exec_prefix = '/home/frontend/altrepo_front/venv'
  sys.platlibdir = 'lib64'
  sys.executable = '/usr/bin/python3'
  sys.prefix = '/home/frontend/altrepo_front/venv'
  sys.exec_prefix = '/home/frontend/altrepo_front/venv'
  sys.path = [
    '/home/frontend/altrepo_front/venv/lib64/python39.zip',
    '/home/frontend/altrepo_front/venv/lib64/python3.9',
    '/home/frontend/altrepo_front/venv/lib64/python3.9/lib-dynload',
  ]
Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding
Python runtime state: core initialized
ModuleNotFoundError: No module named 'encodings'

Current thread 0x00007f2ea7b0f380 (most recent call first):
<no Python frame>
2021/10/27 10:10:40 [notice] 6923#6923 process 10536 exited with code 1
2021/10/27 10:10:40 [warn] 6927#6927 failed to start application "frontend"
2021/10/27 10:10:40 [alert] 6927#6927 failed to apply new conf
RiderALT commented 2 years ago

oops. it's my mistake. There was no access from unit process to the path of venv.conf

But the log is very unformative and it is not clear what you need .

mar0x commented 2 years ago

@RiderALT could you please share the details. It is very common error and would be good to let other people to know more about the reason and the way how to avoid it.

RiderALT commented 2 years ago

this code in src/python/nxt_python.c:nxt_python_start() check access to venv.conf if there is no access, then pep405 mode is not enabled.

        nxt_memcpy(path, c->home, len);
        nxt_memcpy(path + len, pyvenv, sizeof(pyvenv));

        pep405 = (access(path, R_OK) == 0);

        nxt_free(path);

next, 

        if (pep405) {
            mbstowcs(nxt_py_home, c->home, len);
            mbstowcs(nxt_py_home + len, bin_python, sizeof(bin_python));
            Py_SetProgramName(nxt_py_home);

        } else {
            mbstowcs(nxt_py_home, c->home, len + 1);
            Py_SetPythonHome(nxt_py_home);
        }

if pep405 is false, unit exports PYTHONHOME environment variable and python3.9 fall with the 'encodings' error.

If you add an entry to the log that there is no access to venv.conf file in home path and pep405 is not activated - it would be much easier and faster to find out the causes of fall.

mar0x commented 2 years ago

@RiderALT Thanks for the explanation.

comradeFreeman commented 2 years ago

@RiderALT, can You please explain more detailed according to described situation in post? What do You need to do with PYTHONHOME to avoid Unit error? Maybe something else? I'm facing the same problem, but can do nothing with it unfortunately even having solution :(

RiderALT commented 2 years ago

@comradeFreeman use gdb to find. I am moved from unit to another solution for a reasons not related to errors in unit.

mar0x commented 2 years ago

@comradeFreeman No need to set PYTHONHOME environment variable, instead setup virtual environment path via home option in your application configuration.

VBart commented 2 years ago

@RiderALT If your reasons are related to some functionality missing in Unit or something is better in some other product, then it would be nice if you can share these reasons. Thanks.

RiderALT commented 2 years ago

@VBart configuration scheme without files in /etc and not obvious rpm packaging scheme for applications - first reason. #501 - another one. And really, i spent a ton of time to search for the reason not to start the Python application - logs are not very informative in my case with python.

But unit is built for ALT Linux https://packages.altlinux.org/en/sisyphus/srpms/unit/ and I look at its development.

vgiannokostas commented 2 years ago

Do you have any updates on that issue ? I am facing the exact same problem

mar0x commented 2 years ago

@vgiannokostas Please share the details of your environment:

In short, you need to verify the Python version configured for application and used in virtual environment (should be the same), check the presence and access right for pyvenv.cfg file at the root of virtual environment, avoid Python-related environment variables when starting Unit.

MrGecko commented 2 years ago

How i fixed it on Manjaro:

I edited my systemd unit.service script and manually specified --user and --group parameters since I couldn't get it working with the default values specified by unitd -h (in my case it asked for 'nobody' as user and user's primary group as group). Then I did the correct chown -R user:group . in my app folder

vgiannokostas commented 2 years ago

@mar0x Thank you for your reply. I hope my answers below can help you understand more

  1. Exact OS
Distributor ID: Ubuntu
Description:    Ubuntu 20.04 LTS
Release:    20.04
Codename:   focal

  1. Unit version

    unit version: 1.26.1
configured as ./configure --prefix=/usr --state=/var/lib/unit --control=unix:/var/run/control.unit.sock --pid=/var/run/unit.pid --log=/var/log/unit.log 
--tmp=/var/tmp --user=unit --group=unit --tests --openssl --modules=/usr/lib/unit/modules --libdir=/usr/lib/x86_64-linux-gnu --cc-opt='-g -O2 
-fdebug-prefix-map=/data/builder/debuild/unit-1.26.1/pkg/deb/debuild/unit-1.26.1=. -specs=/usr/share/dpkg/no-pie-compile.specs 
-fstack-protector-strong -Wformat -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fPIC' --ld-opt='-Wl,-Bsymbolic-functions 
-specs=/usr/share/dpkg/no-pie-link.specs -Wl,-z,relro -Wl,-z,now -Wl,--as-needed -pie'

  2. Unit installation method I followed these instructions here

  3. I start unit like:

    sudo systemctl restart unit

  4. ENV

    SHELL=/bin/bash
PWD=/home/ubuntu/mainly-ui/venv
LOGNAME=ubuntu
XDG_SESSION_TYPE=tty
MOTD_SHOWN=pam
HOME=/home/ubuntu
LANG=C.UTF-8
LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.zst=01;31:*.tzst=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.wim=01;31:*.swm=01;31:*.dwm=01;31:*.esd=01;31:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36:
VIRTUAL_ENV=/home/ubuntu/mainly-ui/venv
LC_TERMINAL=iTerm2
SSH_CONNECTION=85.24.210.114 52919 172.26.5.212 22
LESSCLOSE=/usr/bin/lesspipe %s %s
XDG_SESSION_CLASS=user
TERM=xterm-256color
LESSOPEN=| /usr/bin/lesspipe %s
USER=ubuntu
LC_TERMINAL_VERSION=3.4.15
SHLVL=1
XDG_SESSION_ID=1062
LC_CTYPE=C.UTF-8
XDG_RUNTIME_DIR=/run/user/1000
PS1=(venv) \[\e]0;\u@\h: \w\a\]${debian_chroot:+($debian_chroot)}\[\033[01;32m\]\u@\h\[\033[00m\]:\[\033[01;34m\]\w\[\033[00m\]\$
SSH_CLIENT=85.24.210.114 52919 22
XDG_DATA_DIRS=/usr/local/share:/usr/share:/var/lib/snapd/desktop
PATH=/home/ubuntu/mainly-ui/venv/bin:/home/ubuntu/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1000/bus
SSH_TTY=/dev/pts/0
OLDPWD=/home/ubuntu/mainly-ui
_=/usr/bin/printenv

  5. Application configuration:

    {
"listeners": {
    "127.0.0.1:80": {
        "pass": "routes"
    },

    "*:443": {
        "pass": "routes",
        "tls": {
            "certificate": "certbot2"
        }
    }
},
"routes": [ {
  "match": {
    "uri": [
        "/api/*",
        "/docs",
        "/docs/*",
        "/openapi.json"
     ]
    },
  "action": {
      "pass": "applications/miranda_webservices"
    }
},
{
  "match": {
      "uri" : "*"
  },
  "action": {
      "share" : "/home/ubuntu/mainly-ui/frontend/out/$uri",
        "fallback": {
            "share": "/home/ubuntu/mainly-ui/frontend/out/index.html"
        }
  }
}
],

"applications": {
    "miranda_webservices": {
        "type": "python",
        "path": "/home/ubuntu/mainly-ui/webservice/",
        "home": "/home/ubuntu/mainly-ui/venv",
        "module": "main",
        "protocol": "asgi",
        "callable": "app",
        "environment": {
            "WEBSERVICE_CONFIG_FILE": "/home/ubuntu/webservice_config.yml",
            "MIRANDA_LOGFILE": "/home/ubuntu/miranda_log.txt"
        }
    }
},

"access_log": "/var/log/access.log"
}

  6. Error message (from /var/log/unit.log)

2022/03/03 14:27:02 [info] 176019#176019 "miranda_webservices" prototype started
2022/03/03 14:27:02 [info] 176020#176020 "miranda_webservices" application started
Python path configuration:
  PYTHONHOME = '/home/ubuntu/mainly-ui/venv'
  PYTHONPATH = (not set)
  program name = 'python3'
  isolated = 0
  environment = 1
  user site = 1
  import site = 1
  sys._base_executable = '/usr/bin/python3'
  sys.base_prefix = '/home/ubuntu/mainly-ui/venv'
  sys.base_exec_prefix = '/home/ubuntu/mainly-ui/venv'
  sys.executable = '/usr/bin/python3'
  sys.prefix = '/home/ubuntu/mainly-ui/venv'
  sys.exec_prefix = '/home/ubuntu/mainly-ui/venv'
  sys.path = [
    '/home/ubuntu/mainly-ui/venv/lib/python38.zip',
    '/home/ubuntu/mainly-ui/venv/lib/python3.8',
    '/home/ubuntu/mainly-ui/venv/lib/python3.8/lib-dynload',
  ]
Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding
Python runtime state: core initialized
ModuleNotFoundError: No module named 'encodings'

Current thread 0x00007f5ff55df940 (most recent call first):
<no Python frame>
2022/03/03 14:27:02 [notice] 176019#176019 app process (isolated 176020) exited with code 1
2022/03/03 14:27:02 [warn] 149010#149010 failed to start application "miranda_webservices"
2022/03/03 14:27:02 [alert] 149010#149010 failed to apply new conf
2022/03/03 14:27:02 [notice] 149005#149005 process 176019 exited with code 0
  1. pyenv.cfg of the virtual environment exists. It's user and group are set to unit.
mar0x commented 2 years ago

@vgiannokostas Thank you for complete setup information.

I suspect it is access rights issue. The application process (which is started as unit user in your case) have no access to /home/ubuntu/mainly-ui/webservice and /home/ubuntu/mainly-ui/venv. To ensure this check the access rights:

$ ls -ld /home/ubuntu /home/ubuntu/mainly-ui /home/ubuntu/mainly-ui/webservice /home/ubuntu/mainly-ui/venv

Consider to add user and group attributes (see the documentation) for miranda_webservices application. This attributes controls the effective user to run the application. No need to change the pyenv.cfg ownership.

mar0x commented 2 years ago

@MrGecko Although the issue can be solved the way you describe, application user and group attributes is more convenient and flexible way to configure effective application process credentials.

tom-friedrichsen commented 2 years ago

Hello all,

it seems I have the same issue, I don't get it. I'm working under Alpine3.15, but tested under Ubuntu 21.10 with the same effects and log entries. I try to get the Python Flask demo application up and running. I use Python 3.9.7. I followed the documentation, still get the error message: 

{
        "error": "Failed to apply new configuration."
}

Logfile entries are:

2022/06/06 09:44:21 [info] 6061#6061 "flask" prototype started
2022/06/06 09:44:21 [info] 6062#6062 "flask" application started
Python path configuration:
  PYTHONHOME = '/root/tmp/venv/'
  PYTHONPATH = (not set)
  program name = 'python3'
  isolated = 0
  environment = 1
  user site = 1
  import site = 1
  sys._base_executable = '/usr/bin/python3'
  sys.base_prefix = '/root/tmp/venv'
  sys.base_exec_prefix = '/root/tmp/venv'
  sys.platlibdir = 'lib'
  sys.executable = '/usr/bin/python3'
  sys.prefix = '/root/tmp/venv'
  sys.exec_prefix = '/root/tmp/venv'
  sys.path = [
    '/root/tmp/venv/lib/python39.zip',
    '/root/tmp/venv/lib/python3.9',
    '/root/tmp/venv/lib/python3.9/lib-dynload',
  ]
Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding
Python runtime state: core initialized
ModuleNotFoundError: No module named 'encodings'

Current thread 0x000055e833d2cd60 (most recent call first):
<no Python frame>
2022/06/06 09:44:21 [notice] 6061#6061 app process (isolated 6062) exited with code 1
2022/06/06 09:44:21 [warn] 5530#5530 failed to start application "flask"
2022/06/06 09:44:21 [alert] 5530#5530 failed to apply new conf
2022/06/06 09:44:21 [notice] 5521#5521 process 6061 exited with code 0

Access rights of the working folder is set to unit:unit

siren:~/tmp# ll
total 8K
drwxr-xr-x    5 unit     unit        4.0K Jun  6 09:33 venv
-rw-r--r--    1 unit     unit         110 Jun  6 09:35 wsgi.py

Unit was started as root, subprocesses are of user unit as expected:

siren:/# ps ax | grep unitd
 5521 root      0:00 {unitd} unit: main v1.26.1 [/usr/sbin/unitd --no-daemon --log /var/log/unit.log --control unix:/run/control.unit.sock]
 5529 unit      0:00 {unitd} unit: controller
 5530 unit      0:00 {unitd} unit: router
 6457 root      0:00 grep unitd
siren:/#

The configuration is used as shown in the documentation: 

siren:~# cat config.json
{
    "listeners": {
        "*:80": {
            "pass": "applications/flask"
        }
    },

    "applications": {
        "flask": {
            "type": "python 3.9.7",
            "path": "/root/tmp/",
            "home": "/root/tmp/venv/",
            "module": "wsgi",
            "callable": "app"
        }
    }
}

Obviously I missed something - any hints for me, please?

Best Tom

MagnusOxlund commented 2 years ago

@tom-friedrichsen Try specifying a user in your configuration:

{
    "listeners": {
        "*:80": {
            "pass": "applications/flask"
        }
    },

    "applications": {
        "flask": {
            "type": "python 3.9.7",
            "path": "/root/tmp/",
            "home": "/root/tmp/venv/",
            "module": "wsgi",
            "callable": "app",
            "user": "siren",
            "group": "siren"
        }
    }
}
tom-friedrichsen commented 2 years ago

@tom-friedrichsen Try specifying a user in your configuration:

{ "listeners": { "*:80": { "pass": "applications/flask" } },

"applications": {
    "flask": {
        "type": "python 3.9.7",
        "path": "/root/tmp/",
        "home": "/root/tmp/venv/",
        "module": "wsgi",
        "callable": "app",
        "user": "siren",
        "group": "siren"
    }
}

}

@MagnusOxlund : Thanks a lot for your reply, and pls accept my appology, that I did not answer to my question as soon as I was able to identify ma mistake. Shame on me, it was a stupid beginner mistake:

Best Tom

lcrilly commented 2 years ago

Try specifying a user in your configuration:

@MagnusOxlund thanks for being an active member of the Unit community! Please contact me and we'll send you some Unit swag: liam@nginx.com

MagnusOxlund commented 2 years ago

@tom-friedrichsen Not at all 🙂 Thank you for sharing the details, I'm sure that will be valuable to someone else. I'm glad you figured it out.