Emperor + venv by pipenv -> Fatal Python error

[tomako]

I use uWSGI in Emperor mode. It has already 2 vassals both are using regular venv. I've added a 3rd one whose venv was generated by pipenv and it doesn't work. uwsgi.ini

[uwsgi]
virtualenv = /home/xxx/.local/share/virtualenvs/ten-XqPi2tsR
chdir = /home/xxx/ten/
wsgi-file = app.py
touch-reload = app.py
socket = 127.0.0.1:9003
master = true
processes = 2
daemonize = /var/log/uwsgi/ten.log

ten.log

*** Starting uWSGI 2.0.15 (64bit) on [Tue Nov 28 03:22:14 2017] ***
compiled with version: 5.4.0 20160609 on 26 April 2017 07:15:54
os: Linux-4.4.0-97-generic #120-Ubuntu SMP Tue Sep 19 17:28:18 UTC 2017
nodename: ai42
machine: x86_64
clock source: unix
detected number of CPU cores: 40
current working directory: /etc/uwsgi/vassals
detected binary path: /usr/local/bin/uwsgi
!!! no internal routing support, rebuild with pcre support !!!
chdir() to /home/xxx/ten
your processes number limit is 257415
your memory page size is 4096 bytes
detected max file descriptor number: 1024
lock engine: pthread robust mutexes
thunder lock: disabled (you can enable it with --thunder-lock)
uwsgi socket 0 bound to TCP address 127.0.0.1:9003 fd 3
Python version: 3.5.2 (default, Sep 14 2017, 22:51:06)  [GCC 5.4.0 20160609]
Set PythonHome to /home/xxx/.local/share/virtualenvs/ten-XqPi2tsR
Fatal Python error: Py_Initialize: Unable to get the locale encoding
ImportError: No module named 'encodings'

Few things what I've tried already:

• The venv works, the app can be started using that venv
• The app also works using uwsgi from command line
• The pipenv, the virtual environment have been reinstalled more times
• Eventually a regular venv has been created and the app works

This error appears mostly when the python environment is corrupted in some way for example wrong PYTHONHOME settings. Seemingly uWSGI manages that well. The most annoying thing is that this config works in my stage environment but doesn't in production while all the components and versions are the same. Well, something must be different but I have no idea what.

[unbit]

Hi @tomako ensure the venv has been built for the same version of the uWSGi binary/plugin (from your logs it results to "Python version: 3.5.2 (default, Sep 14 2017, 22:51:06)"

[tomako]

Sure, it is. That was the first thing what I have checked.

[xrmx]

Could you please an ls of /home/xxx/.local/share/virtualenvs/ten-XqPi2tsR please?

[tomako]

Sure, here it is. ten_pyvenv2.txt

[DanielSwain]

I'm having the same error and posted a StackOverflow question about this last night. Any update on this issue @unbit ?

[DanielSwain]

I've updated my Stackoverflow question with how I worked around this problem.

[demeralde]

I have the same issue :(

[demeralde]

Yep, something weird is going on with uWSGI when it uses the virtualenv.

To test importing encodings with uWSGI, I created a file (test.py) that prints one of the module's functions:

import encodings

print(encodings.search_function)

Running it without the virtualenv works fine:

$ uwsgi --python test.py 
*** Starting uWSGI 2.0.17 (64bit) on [Wed Mar 14 06:18:54 2018] ***
compiled with version: 5.4.0 20160609 on 14 March 2018 04:29:37
os: Linux-4.4.0-116-generic #140-Ubuntu SMP Mon Feb 12 21:23:04 UTC 2018
nodename: b4ubuyhomeinspectionsadelaide
machine: x86_64
clock source: unix
detected number of CPU cores: 1
current working directory: /home/daniel/b4ubuy-website
detected binary path: /usr/local/bin/uwsgi
!!! no internal routing support, rebuild with pcre support !!!
your processes number limit is 3913
your memory page size is 4096 bytes
detected max file descriptor number: 1024
lock engine: pthread robust mutexes
thunder lock: disabled (you can enable it with --thunder-lock)
Python version: 3.5.2 (default, Nov 23 2017, 16:37:01)  [GCC 5.4.0 20160609]
*** Python threads support is disabled. You can enable it with --enable-threads ***
Python main interpreter initialized at 0xa60c30
your mercy for graceful operations on workers is 60 seconds
mapped 72904 bytes (71 KB) for 1 cores
*** Operational MODE: command ***
*** uWSGI is running in multiple interpreter mode ***
spawned uWSGI worker 1 (and the only) (pid: 7691, cores: 1)
<function search_function at 0x7f406c449158>

But when the virtualenv is specified, it can't import encodings:

$ uwsgi --python test.py --home .venv
*** Starting uWSGI 2.0.17 (64bit) on [Wed Mar 14 06:19:20 2018] ***
compiled with version: 5.4.0 20160609 on 14 March 2018 04:29:37
os: Linux-4.4.0-116-generic #140-Ubuntu SMP Mon Feb 12 21:23:04 UTC 2018
nodename: b4ubuyhomeinspectionsadelaide
machine: x86_64
clock source: unix
detected number of CPU cores: 1
current working directory: /home/daniel/b4ubuy-website
detected binary path: /usr/local/bin/uwsgi
!!! no internal routing support, rebuild with pcre support !!!
your processes number limit is 3913
your memory page size is 4096 bytes
detected max file descriptor number: 1024
lock engine: pthread robust mutexes
thunder lock: disabled (you can enable it with --thunder-lock)
Python version: 3.5.2 (default, Nov 23 2017, 16:37:01)  [GCC 5.4.0 20160609]
Set PythonHome to .venv
Fatal Python error: Py_Initialize: Unable to get the locale encoding
ImportError: No module named 'encodings'

Also note that the virtualenv works and I can import encodings in its Python shell.

I assume this suggests the problem is on uWSGI's end.

[DanielSwain]

I just updated my StackOverflow question with another workaround. My first post about a workaround (above) turned out to only work on the command line. I'm still looking for a real solution.

[evcb]

I am facing the same problem. My virtualenv is python3.6, however when I see the error in the logs, it mentions the previous version I had: python3.5.

** Starting uWSGI 2.0.15 (64bit) on [Tue Mar 20 15:37:20 2018] ***
compiled with version: 5.4.0 20160609 on 08 June 2017 14:53:31
os: Linux-4.4.0-116-generic #140-Ubuntu SMP Mon Feb 12 21:23:04 UTC 2018
nodename: web
machine: x86_64
clock source: unix
detected number of CPU cores: 4
current working directory: /etc/uwsgi/sites
detected binary path: /usr/local/bin/uwsgi
!!! no internal routing support, rebuild with pcre support !!!
chdir() to /home/web/sites/sources/atom
your processes number limit is 15646
your memory page size is 4096 bytes
detected max file descriptor number: 1024
lock engine: pthread robust mutexes
thunder lock: disabled (you can enable it with --thunder-lock)
uwsgi socket 0 bound to UNIX address /run/uwsgi/atom.sock fd 3
setuid() to 1001
Python version: 3.5.2 (default, Nov 23 2017, 16:37:01)  [GCC 5.4.0 20160609]
Set PythonHome to /home/web/sites/envs/atom
Fatal Python error: Py_Initialize: Unable to get the locale encoding
ImportError: No module named 'encodings'

On my .ini file I only have the variable home set. This was working fine before I installed python3.6.

[xrmx]

@hellvix how you compiled uwsgi? how are you calling it?

[demeralde]

@hellvix I'm experiencing the same version mismatch, too

[evcb]

@xrmx I have installed it using pip -H install uwsgi and then I run it with a non root user through systemctl:

[Unit]
Description=uWSGI Emperor service

[Service]
ExecStartPre=/bin/bash -c 'mkdir -p /run/uwsgi; chown web:www-data /run/uwsgi'
ExecStart=/usr/local/bin/uwsgi --emperor /etc/uwsgi/sites
Environment="DJANGO_SETTINGS_MODULE=core.settings_stage"
Restart=always
KillSignal=SIGQUIT
Type=notify
NotifyAccess=all

[Install]
WantedBy=multi-user.target

[uwsgi]
project = atom
socket_filename = atom
wsgi_filename = core
env_dir = atom 
uid = web
base = /home/%(uid)

chdir = %(base)/sites/sources/%(project)
home = %(base)/sites/envs/%(env_dir)
module = %(wsgi_filename).wsgi:application
logto = /var/log/uwsgi.log

master = true
processes = 5

socket = /run/uwsgi/%(socket_filename).sock
chown-socket = %(uid):www-data
chmod-socket = 660
vacuum = true

Edit

I managed to resolve the problem by uninstalling uwsgi and installing it again. sudo -H pipN uninstall uwsgi. Also, remove the venv you had been using and create a new one.

Now I know why the problem happened. Gotta remember to reinstall uwsgi and your venv when you change python installations like that.

[xrmx]

@hellvix thanks for checking!

Is everyone good after building uwsgi with their current python version? If so I think we can close this.

[demeralde]

@xrmx Awesome, yep, I've got it working by following what @hellvix did with reinstalling uWSGI.

An important thing to note is I had to use a non-root user, so I created one called web (like @hellvix did) and added it to the www-data group. It didn't work using my default user (who is in the sudo) group.

[evcb]

Yes, I solved my problem. You can close it if no one else is having problems.

PS: A part of the solution is to also remove and reinstall your venv. I have edited my solution answer and included this part.

[ThomasFreedman]

I run into this "locale" issue from time to time, however so infrequently that I don't remember how I resolved it or even if the context is the same as before.

I'll document the issue for THIS time.

I'm doing this in a docker context built on top of the Ubuntu official ubuntu:xenial OS image (v16.04). I noticed there are 2 folders for python3* in /usr/lib: 1) a /usr/lib/python3 folder containing ONLY a dist-packages sub folder, and 2) a /usr/lib/python3.5 folder with tons of stuff, looks more like a typical python3 installation.

I also see python3 in /usr/local/lib/python3.5, which contains the pip3 packages I installed.

I can run one of the python3 programs I installed with pip3 from command line, no issues. When I try to run an app from uwsgi I get the locale error.

My uwsgi ini file is pretty basic. This works in a non-docker context using virtualenv, however I don't see much benefit in using venv in docker, so I commented out the virtualenv line in ini file. I have also tried setting the value of virtualenv = to my app package root and to just the python3.5 root. The later for both /usr/local/lib/python3.5 and /usr/lib/python3.5. The output of uwsgi upon starting shows it is setting PythonHome to a python3.5 correlated to the virtualenv = setting.

Here is my uwsgi ini file:

[uwsgi]
uid = bts_tools
gid = www-data
socket = /home/bts_tools/.bts_tools/uwsgi/socket
chmod-socket = 666
# plugin = python3
# virtualenv = /home/bts_tools/.virtualenvs/bts_tools    <--- for non-docker context
# virtualenv = /usr/local/lib/python3.5/dist-packages/bts_tools
# virtualenv = /usr/lib/python3.5/
virtualenv = /usr/local/lib/python3.5
enable-threads = true
lazy-apps = true
workers = 1
module = bts_tools.wsgi
callable: application

And this is how I run uwsgi:

#!/bin/bash
# export PYTHONHOME=/usr/local/lib/python3.5/
# export PYTHONPATH=/usr/local/lib/python3.5/
uwsgi --ini /home/bts_tools/.bts_tools/uwsgi/bts_tools.ini >> /home/bts_tools/.bts_tools/monitor.log 2>&1 &

This issue is such a PITA! There are sooooooooo many people that have it I see in google searches. Often people resolve it via setting the pythonhome or pythonpath, but many times that doesn't work, as in this current docker context.

I installed uwsgi with pip3, as I did my bts_tools app & related programs. I used apt-get to install python-dev python3 python3-dev python3-pip in that order, as I do on a non-docker OS.

Also, can't find much info about uwsgi python plugin. Is that not part of a typical uwsgi installation? Is such a plugin only available if you compile uwsgi source code? What does it do, is it a legacy thing for older uwsgi versions? I have had to comment out that line in my uwsgi ini file as it can't find any plugin named pthon, python3, python3x on any Debian or Ubuntu platform.

[LiamK]

I also ran into this problem. Uwsgi emperor doesn't seem to play nicely with virtualenv. Since the whole point of using a virtualenv is that they are independent of the system python version, and having multiple different versions, this is a pretty severe limitation for emperor. I'm falling back on launching multiple systemd process, which is a pity because emperor looks useful. It would be nice to have this fixed, though.