pypy_setup.py incompatible with PyPy3

[kravietz]

Using PyPy3 which I compiled on my own to get the shared library:

$ pypy
Python 3.2.5 (b2091e973da69152b3f928bfaabd5d2347e6df46, Mar 12 2015, 23:25:39)
[PyPy 2.4.0 with GCC 4.9.2] on linux2

Uwsgi installs without any problems, but won't run:

$ uwsgi --http-socket :9090 --pypy-home ~/pypy3-2.4.0-src
*** Starting uWSGI 2.0.9 (64bit) on [Fri Mar 13 12:07:29 2015] ***
compiled with version: 4.9.2 on 13 March 2015 12:05:46
os: Linux-3.13.0-40-generic #69-Ubuntu SMP Thu Nov 13 17:53:56 UTC 2014
nodename: friedrich
machine: x86_64
(...)
uwsgi socket 0 bound to TCP address :9090 fd 3
debug: OperationError:
debug:  operror-type: SyntaxError
debug:  operror-value: ('invalid syntax', ('c callback', 332, 14, '        print "PyPy WARNING: unable to load paste.script.util.logging_config"\n', 0))

Apparently, the built-in pypy_setup.py file is in Python2 syntax. I fetched it from master and fixed the print()s into the Python3 syntax, but it still crashes with another error:

uwsgi socket 0 bound to TCP address :9090 fd 3
debug: OperationError:
debug:  operror-type: AttributeError
debug:  operror-value: 'module' object has no attribute 'stdout'

Note sure where this one comes from so just filing this report.

[vthorsteinsson]

I have the exact same problem using the latest dev build, 2.6.0 of PyPy3 (which includes the shared library by default, btw).

[wbyoung]

I spent some time today trying to track this down, but was quite unsuccessful. I have reported issue #2430 with pypy as no matter what I try, print seems to be broken when using embedding techniques.

Just a note, @kravietz: the issues with Python3 syntax have been resolved in the pypy setup in the master branch.

From what I learned today, though, it appears that @unbit and the pypy team worked together to create an embedding interface specifically for uWSGI, so perhaps this will be one of the first places to focus for pypy3's embedding support.

[vthorsteinsson]

FYI, and FWIW, I gave up on uWSGI for this reason, but Gunicorn works fine with pypy3 using gevent. I run it as my WSGI middleware for Flask under nginx with no problems and good performance.

[RobertoPrevato]

I had the same problem, trying to setup a Docker container to serve applications with uWSGI, Gevent and pypy3. Since I saw that last long term release still has calls to print without parenthesis, but the master branch in GitHub has a commit to resolve this issue; I tried to:

• clone the repository locally, create manually a package .tar.gz
• pip install this tar.gz directly (also using pip3 explicitly)

It still doesn't work, but with this Python-version related error:

debug: OperationError:
debug:  operror-type: TypeError
debug:  operror-value: startswith first arg must be bytes or a tuple of bytes, not str

It's a pity that it doesn't work, and it's a pity that uWSGI documentation doesn't state clearly that pypy3 is not supported. Documentation is otherwise great and this is like a stain on a white wall.

[amars1]

I installed latest uwsgi using pip and downloaded latest pypy3 pypy3-v5.10.0-linux64.tar.bz2 facing same issue as - debug: OperationError: debug: operror-type: SyntaxError debug: operror-value: ("Missing parentheses in call to 'print'", ('c callback', 332, 14, ' print "PyPy WARNING: unable to load paste.script.util.logging_config"\n', 0))

Nobody bothered to solve this issue? Atleast leave some notes stating this issue so that nobody tries pypy3.

[ghost]

I got it working with the latest PyPy version from portable-pypy. Only a few modifications were needed to convert various strings to bytes.

This is not production ready, probably needs some cleanup. I'm only using PyPy for benchmark purposes.

Based on pypy_setup.py from the uwsgi sources: WIP

[ihor-nahuliak]

I have the same issue

CMD ["./env-pypy3/bin/uwsgi", "--chdir=/var/www/web/", "--pypy-home=./env-pypy3/", "--pypy-lib=./env-pypy3/bin/libpypy3-c.so", "--pypy-wsgi-file=app/application.py", "--http-socket=:5005", "--master", "--processes=1", "--enable-threads", "--threads=2"]

debug: OperationError:
debug:  operror-type: SyntaxError
debug:  operror-value: ("Missing parentheses in call to 'print'", ('c callback', 332, 14, '        print "PyPy WARNING: unable to load paste.script.util.logging_config"\n', 0))

[ghost]

My bad, I should have added to use --pypy-setup uwsgi_pypy_setup.py (uwsgi_pypy_setup.py being the pastebin link given in my previous post)

[ihor-nahuliak]

Could you include this file into the uwsgi library like [pypy] extension?

[PAStheLoD]

FYI for others, the aforementioned uwsgi pypy loader needs to be used this way to work:

export PYTHONPATH=/opt/pypy/venv/site-packages

cd /opt/some-site
wget https://pastebin.com/raw/2Pp55zSW -O uwsgi_pypy_setup.py

/opt/pypy/venv/bin/uwsgi  --pypy-home /opt/pypy/venv/ --pypy-setup ./uwsgi_pypy_setup.py --pypy-lib /opt/pypy/venv/bin/libpypy3-c.so --pypy-wsgi wsgi_file --master --workers 2 --http 0.0.0.0:80 --enable-threads  --so-keepalive --http-keepalive

I tried pypy-wsgi-file, but that did not work for some reason. And specifying --need-app again results in failure.

So, after a bit of meddling with PYTHONPATH uwsgi works with Pypy 3.5 (5.10.1 portable), but it's slower than CPython. (At least on a simple wrk benchmark of serving a static Pyramid site from a Jinja template.)

[jkozera]

Since the pastebin mentioned above has now expired, I've pushed my version of pypy_setup.py to https://github.com/operasoftware/uwsgi-pypy-python3 - hope it helps someone. :)

[gralance]

@jkozera @PAStheLoD In pypy3.5 (7.0), the https://github.com/operasoftware/uwsgi-pypy-python3 and use pypy_setup.py I got error message:

*** Operational MODE: preforking+threaded ***
From cffi callback <function uwsgi_pypy_loader at 0x000055ebcb38e0c0>:
Traceback (most recent call last):
  File "c callback", line 317, in uwsgi_pypy_loader
TypeError: expected string, got bytes object
*** no app loaded. going in full dynamic mode ***
*** uWSGI is running in multiple interpreter mode ***
spawned uWSGI master process (pid: 10090)
spawned uWSGI worker 1 (pid: 10103, cores: 2)
spawned uWSGI worker 2 (pid: 10105, cores: 2)
From cffi callback <function uwsgi_pypy_wsgi_handler at 0x000055ebcb439920>:
Traceback (most recent call last):
  File "c callback", line 502, in uwsgi_pypy_wsgi_handler
TypeError: 'NoneType' object is not callable
[pid: 10105|app: -1|req: -1/1] 123.193.3.10 () {44 vars in 899 bytes} [Thu Feb 21 10:11:38 2019] GET / => generated 0 bytes in 3 msecs (HTTP/2.0 500) 0 headers in 0 bytes (0 switches on core 0)
From cffi callback <function uwsgi_pypy_wsgi_handler at 0x000055ebcb439920>:
Traceback (most recent call last):
  File "c callback", line 502, in uwsgi_pypy_wsgi_handler
TypeError: 'NoneType' object is not callable

[PatTheMav]

@gralance IIRC this error usually happens when uWSGI doesn't know which app/module to load. What's your ini?

[gralance]

Same ini works in pypy2

[uwsgi]
# You should set chdir to the folder of manage.py
pypy-wsgi = django_proj.wsgi:application
master = 1
processes = %k
threads = 2
pypy-home = /usr/local/lib/pypy3  # /usr/local/lib/pypy2 for older version

[PAStheLoD]

Does your INI work with CPython3 properly? It seems the imported module does not have an "application" exported variable. You could try to put debug statements around line 314, to dump the defined and available symbols (variables) of the module. (This will help to confirm whether the right module got imported at all.)

[PatTheMav]

In my tests, PyPy required less parameters to work than PyPy3. To make it work with the linked pypy_setup.py I had to set pypy-wsgi-file, pypy-lib and pypy-setup as well as module to make it all work.

[gralance]

@PatTheMav , thank you so much, finally here is the ini that works

[uwsgi]
# You should set chdir to the folder of manage.py
pypy-wsgi = django_proj.wsgi:application
master = 1
processes = %k
threads = 2
pypy-home = /usr/local/lib/pypy3
pypy-wsgi-file = /<project dir>/django_proj/wsgi.py
pypy-lib = /usr/local/lib/pypy3/bin/libpypy3-c.so
pypy-setup = /<project dir>/django_proj/uwsgi/pypy_setup.py

I missed the pypy-wsgi-file.

[xrmx]

It would be nice if anyone would like to update uwsgi-docs with your findings

[gralance]

@xrmx https://github.com/unbit/uwsgi-docs/compare/master...gralance:patch-1

[johnthagen]

It would be nice if PyPy3 were supported out of the box. For one thing, CPython 2.7 is EOL at the end of this year so Django and the rest of the ecosystem are already beginning to drop support for Python 2. This makes PyPy(2) less and less attractive.

Following the current docs: https://uwsgi-docs.readthedocs.io/en/latest/PyPy.html

Environment:

• Ubuntu 18.04 x64
• PyPy3.6-7.1.1: https://bitbucket.org/squeaky/portable-pypy/downloads/pypy3.6-7.1.1-beta-linux_x86_64-portable.tar.bz2

I create a venv:

$ pypy3 -m venv venv
$ source venv/bin/activate
(venv) $ pip install --upgrade pip
(venv) $ pip install uwsgi

This part of the docs:

The “downside” of this approach is that you need to inform uWSGI where your PyPy installation is at runtime (unless you installed uwsgi via pip or with the setup.py script, in such a case the home will be found automatically)

makes me think that if I install with pip from a PyPy3 venv that everything should be good to go.

But when it's run:

(venv) $ uwsgi --ini nginx/uwsgi.ini
...
unable to load pypy library: libpypy-c.so: cannot open shared object file: No such file or directory

The ini file I used works with CPython 3.6 installed by default in Ubuntu.

[johnthagen]

Tested with portable PyPy3.6-7.1.1, Django 2.2.1, and uwsgi 2.0.18:

Follow this guide to copy the patched pypy_setup.py into your project.

Use the following ini (uwsgi/uwsgi_pypy3.ini):

[uwsgi]
chdir = /<project dir>/

## Process-related settings
master = true
processes = 10
threads = 2
socket = 127.0.0.1:8001

## PyPy3 related settings
pypy-wsgi = <project name>.wsgi:application
pypy-home = /<pypy install folder>/pypy3.6-7.1.1-beta-linux_x86_64-portable
pypy-wsgi-file = /<project dir>/<project name>/wsgi.py
pypy-lib = /<pypy install folder>/pypy3.6-7.1.1-beta-linux_x86_64-portable/bin/libpypy3-c.so

# File taken from https://github.com/unbit/uwsgi-docs/pull/440
pypy-setup =  /<project dir>/uwsgi/pypy_setup.py

$ pypy3 -m venv venv
$ source venv/bin/activate
(venv) $ pip install -U pip

# Install Django, uwsgi, and other requirements into PyPy3 venv.
(venv) $ pip install -r requirements.txt
(venv) $ PYTHONPATH=/<project dir>/venv/site-packages uwsgi --ini uwsgi/uwsgi_pypy3.ini

Hope this helps someone.

Having this be more streamlined would be super helpful.

[weykent]

soo idk who's following this or who cares about it or where the code will end up ultimately, but the most recent uwsgi-pypy-python3 i found had some errors in it, which i corrected. code lives here: https://github.com/weykent/uwsgi-pypy-python3

[owen800q]

soo idk who's following this or who cares about it or where the code will end up ultimately, but the most recent uwsgi-pypy-python3 i found had some errors in it, which i corrected. code lives here: https://github.com/weykent/uwsgi-pypy-python3

I have this issue on pypy2.7 created by venv with Flask app.py

app = Flask(__name__)
if __name__=='__main__':
    app.run()

uwsgi.py

from app import app as application

if __name__ == '__main__':
    application.run()

uwsgi --socket 127.0.0.1:5000 --processes 4 --threads 8 --protocol=ht --pypy-lib ./venv/bin/libpypy3-c.so --pypy-wsgi uwsgi --pypy-setup ./pypy_setup.py 

*** Starting uWSGI 2.0.18 (64bit) on [Thu Sep 19 16:43:19 2019] ***
compiled with version: 7.4.0 on 19 September 2019 04:28:50
os: Linux-4.15.0-62-generic #69-Ubuntu SMP Wed Sep 4 20:55:53 UTC 2019
nodename: kwan-LIFEBOOK-U536
machine: x86_64
clock source: unix
pcre jit disabled
detected number of CPU cores: 4
current working directory: /media/Programming/Contributions/OpenAcg
detected binary path: /media/Programming/Contributions/OpenAcg/venv/bin/uwsgi
*** WARNING: you are running uWSGI without its master process manager ***
your processes number limit is 15150
your memory page size is 4096 bytes
detected max file descriptor number: 1048576
lock engine: pthread robust mutexes
thunder lock: disabled (you can enable it with --thunder-lock)
uwsgi socket 0 bound to TCP address 127.0.0.1:5000 fd 3
Initialized PyPy with Python 3.6.1 (784b254d669919c872a505b807db8462b6140973, Apr 16 2019, 18:18:28)
[PyPy 7.1.1-beta0 with GCC 8.2.0]
PyPy Home: /media/Programming/Contributions/OpenAcg/venv
your server socket listen backlog is limited to 100 connections
your mercy for graceful operations on workers is 60 seconds
mapped 584448 bytes (570 KB) for 32 cores
*** Operational MODE: preforking+threaded ***
From cffi callback <function uwsgi_pypy_loader at 0x00007f920fe72200>:
Traceback (most recent call last):
  File "c callback", line 263, in uwsgi_pypy_loader
AttributeError: module 'uwsgi' has no attribute 'application'
*** no app loaded. going in full dynamic mode ***
*** uWSGI is running in multiple interpreter mode ***
spawned uWSGI worker 1 (pid: 23780, cores: 8)
spawned uWSGI worker 2 (pid: 23783, cores: 8)
spawned uWSGI worker 3 (pid: 23784, cores: 8)
spawned uWSGI worker 4 (pid: 23785, cores: 8)

Once I change the pypy_setup.py file to the one you corrected, the problem solved. Thanks a lot..

[johnthagen]

For what it's worth, we ended up just switching to Gunicorn. The configuration is much simpler when trying to support PyPy3, and Gunicorn runs PyPy3 on their CI now, which provided us a lot more confidence than relying on third party configuration scripts listed in this thread.

[PixyMisa]

Thanks everyone for the detailed discussion and the patches.

Can confirm that using uWSGI installed with pip under PyPy 2 with the new pypy_setup.py file (and the right parameters to point to the PyPy 3 directory and .so) worked for my application.