provide new python version for uWSGI-compatible PyPy?

[requiredfield]

/cc @unbit

I wanted to install and run uWSGI in a pypy virtualenv, so I did

pyenv install pypy-2.4.0
pyenv virtualenv pypy-2.4.0 myenv
pyenv activate myenv
pip install uwsgi

But the uWSGI version that was installed didn't end up working (failed with "unable to load pypy library: libpypy-c.so" IIRC). I found some relevant docs here and here but am still unsure what the simplest set of steps I need to take is to get this working, and thought asking here could result in helping others too. For example, do I pyenv install pypy-2.4.0-src then build PyPy myself with --shared? (If I misunderstand how pyenv's *-src python versions work, could you please point me to those docs?) Would it be possible/helpful for pyenv to provide a new python version for a uWSGI-compatible pypy out of the box? Thanks in advance for any help.

[requiredfield]

Hi @yyuu, just thought I'd try pinging you on this in case it got lost in the shuffle. Any advice on how to install a --shared build of pypy via pyenv? Much appreciate any tips.

[yyuu]

The PyPy project does not provide binary distribution built with --shared. If you want one, please build PyPy with using pypy-*-src (e.g. pypy-2.4.0-src) with special parameters.

And, python-build can read the build parameters of PyPy on the compilation. Please try command like env PYPY_OPTS="-Ojit --shared --gcrootfinder=shadowstack pypy/goal/targetpypystandalone pyenv install -v pypy-2.4.0-src to build PyPy with --shared.

https://github.com/yyuu/pyenv/blob/ff995b66548b00d7ef878a5b6e18b128d0e08e8f/plugins/python-build/bin/python-build#L815

[yyuu]

Building PyPy 2.4.0 with --shared requires to set LD_LIBRARY_PATH (or DYLD_LIBRARY_PATH on Darwin) on runtime. I'll try to find a way to set rpath for the generated binary like python-build is doing for CPython.

[requiredfield]

Thank you for the tips, @yyuu! I'm actually about halfway through building pypy after following your instructions above (just had to add the missing " after pypy/goal/targetpypystandalone).

I'll keep an eye out for updates on the LD_LIBRARY_PATH issue. If I understand correctly, for now I can just set it in /etc/environment (for example) to make sure pypy will work, is that right? What would I set it to?

Thanks again for all your help.

[yyuu]

It takes very very long time to build PyPy....

IIRC, on Linux, LD_LIBRARY_PATH is just an environment variable for /lib/ld-linux.so. You can set the environment variables anywhere you can set it, including /etc/environment and your ~/.bashrc.

[requiredfield]

It takes very very long time to build PyPy....

No kidding. After going strong for 92 minutes, my build just failed with a MemoryError :cry:

...
[1eae1] translation-task}
[Timer] Timings:
[Timer] annotate                       ---  529.0 s
[Timer] rtype_lltype                   --- 2151.4 s
[Timer] pyjitpl_lltype                 --- 1886.2 s
[Timer] backendopt_lltype              ---  299.5 s
[Timer] stackcheckinsertion_lltype     ---  437.0 s
[Timer] database_c                     ---  249.8 s
[Timer] ===========================================
[Timer] Total:                         --- 5552.8 s
[translation:info] Error:
[translation:info]    File "/tmp/python-build.20141010053517.10630/pypy-pypy-c6ad44ecf5d8/rpython/translator/goal/translate.py", line 316, in main
[translation:info]     drv.proceed(goals)
[translation:info]    File "/tmp/python-build.20141010053517.10630/pypy-pypy-c6ad44ecf5d8/rpython/translator/driver.py", line 531, in proceed
[translation:info]     return self._execute(goals, task_skip = self._maybe_skip())
[translation:info]    File "/tmp/python-build.20141010053517.10630/pypy-pypy-c6ad44ecf5d8/rpython/translator/tool/taskengine.py", line 114, in _execute
[translation:info]     res = self._do(goal, taskcallable, *args, **kwds)
[translation:info]    File "/tmp/python-build.20141010053517.10630/pypy-pypy-c6ad44ecf5d8/rpython/translator/driver.py", line 276, in _do
[translation:info]     res = func()
[translation:info]    File "/tmp/python-build.20141010053517.10630/pypy-pypy-c6ad44ecf5d8/rpython/translator/driver.py", line 429, in task_database_c
[translation:info]     database = cbuilder.build_database()
[translation:info]    File "/tmp/python-build.20141010053517.10630/pypy-pypy-c6ad44ecf5d8/rpython/translator/c/genc.py", line 177, in build_database
[translation:info]     db.complete()
[translation:info]    File "/tmp/python-build.20141010053517.10630/pypy-pypy-c6ad44ecf5d8/rpython/translator/c/database.py", line 306, in complete
[translation:info]     add_dependencies(node.enum_dependencies(), node)
[translation:info]    File "/tmp/python-build.20141010053517.10630/pypy-pypy-c6ad44ecf5d8/rpython/translator/c/database.py", line 294, in add_dependencies
[translation:info]     self.get(value, parent and parent._funccodegen_owner)
[translation:info]    File "/tmp/python-build.20141010053517.10630/pypy-pypy-c6ad44ecf5d8/rpython/translator/c/database.py", line 226, in get
[translation:info]     node = self.getcontainernode(container)
[translation:info]    File "/tmp/python-build.20141010053517.10630/pypy-pypy-c6ad44ecf5d8/rpython/translator/c/database.py", line 158, in getcontainernode
[translation:info]     self.gctransformer.consider_constant(T, container)
[translation:info]    File "/tmp/python-build.20141010053517.10630/pypy-pypy-c6ad44ecf5d8/rpython/memory/gctransform/framework.py", line 495, in consider_constant
[translation:info]     self.layoutbuilder.consider_constant(TYPE, value, self.gcdata.gc)
[translation:info]    File "/tmp/python-build.20141010053517.10630/pypy-pypy-c6ad44ecf5d8/rpython/memory/gctypelayout.py", line 421, in consider_constant
[translation:info]     self.iseen_roots[value] = True
[translation:info]    File "/tmp/python-build.20141010053517.10630/pypy-pypy-c6ad44ecf5d8/rpython/tool/identity_dict.py", line 21, in __setitem__
[translation:info]     self._dict[id(arg)] = val
[translation:ERROR] MemoryError

I've got 2GB of memory on this i686 machine and according to http://pypy.readthedocs.org/en/latest/getting-started-python.html that should be enough. Any suggestions that don't require sacrificing -Ojit? /cc @fijal

Thanks in advance for any help.

[requiredfield]

If I switch to an Ubuntu Trusty x86_64 machine, is there some way I can use the libpypy-c-x86_64.so linked from http://uwsgi-docs.readthedocs.org/en/latest/PyPy.html with PyEnv to avoid having to build PyPy myself?

[yyuu]

I added some workaround in pypy_builder in python-build at 42bb40a5d837451e203c068dfa3d1c3596ac8ede. This allows you to build shared PyPy by running env PYPY_OPTS='-Ojit --shared --gcrootfinder=shadowstack' pyenv install -v pypy-2.4.0-src. The build problem of PyPy with --shared must be cleared with this fix.

I'm not sure how the built binary of libpypy-c.so on http://uwsgi-docs.readthedocs.org/en/latest/PyPy.html is being maintained. Plus, what the version of PyPy is. I would recommend you to build PyPy yourself if you want to maintain your application built on the libpypy-c.so. Anyway, please decide yourself how you maintain your application.

[requiredfield]

Now I'm getting [translation:ERROR] Skipped: no _curses or _minimal_curses module. When I import curses by hand with a pyenv python I get ImportError too, though with a system python it works. Looks like other users have had this problem with pyenv too: #240

[yyuu]

RPython (which is a toolchain of PyPy) requires existing CPython installation, you need to have CPython with proper modules before start building PyPy. The _curses error means your Python installation lacks curses. Please install curses headers (e.g. libncurses5-dev on Debian) and rebuild CPython and try building PyPy again.

Because I am not a PyPy guy, please ask PyPy guys how to build PyPy from source.

[requiredfield]

Should have said I already have libncurses5-dev installed. And like I said when I import _curses by hand it works:

> pyenv versions
  system
* pypy-2.4.0 (set by /home/ubuntu/.pyenv/version)
> python -c 'import _curses'  # no ImportError
> pyenv shell system
> python -c 'import _curses'  # no ImportError
> env PYPY_OPTS='-Ojit --shared --gcrootfinder=shadowstack' pyenv install -v pypy-2.4.0-src
...
[translation:ERROR] Skipped: no _curses or _minimal_curses module
...

[yyuu]

Sorry I just overlooked about curses. I just pushed a patch. Please try again with the Python version which has curses module in environment PYENV_RPYTHON_VERSION and try building PyPy again.

On Monday, October 13, 2014, requiredfield <notifications@github.com javascript:_e(%7B%7D,'cvml','notifications@github.com');> wrote:

Should have said I already have libncurses5-dev installed. And like I said when I import _curses by hand it works:

pyenv versions system

• pypy-2.4.0 (set by /home/ubuntu/.pyenv/version) python -c 'import _curses' # no ImportError pyenv shell system python -c 'import _curses' # no ImportError env PYPY_OPTS='-Ojit --shared --gcrootfinder=shadowstack' pyenv install -v pypy-2.4.0-src ... [translation:ERROR] Skipped: no _curses or _minimal_curses module ...

— Reply to this email directly or view it on GitHub https://github.com/yyuu/pyenv/issues/244#issuecomment-58810440.

山下 優 (やました ゆう) peek824545201@gmail.com

[requiredfield]

Because this wasn't working with pyenv, I already went ahead and built pypy with --shared without pyenv on this machine. But I can try again with pyenv on another machine later and let you know if it works.

[requiredfield]

From http://lists.unbit.it/pipermail/uwsgi/2014-October/007650.html:

I need to ask you (and others people wanting to try uWSGI+pypy) to wait a couple of weeks as Maciej Fijałkowski (pypy core developer) is working on having libpypy built-in by default in official pypy binaries (it is a sponsored work, so it should be done fast). This will solve a lot of issues and will allow us to work better (and faster) on the plugin (the objective is having it on par as CPython before 2.1).

Wait until that's done and then update pyenv with support?

[yyuu]

If the next official pypy binary will come with libpypy.so, pyenv will also include it because we don't modify original distributions basically (we may apply additional patches only if the original distribution is not working). Please wait for a while until the next release of pypy....

[nxsofsys]

Seems env PYPY_OPTS='-Ojit --shared --gcrootfinder=shadowstack' pyenv install -v pypy-2.4.0-src is the only way to build pypy on debian wheezy, pyenv install -v pypy-2.4.0-src fails with error: mv: cannot statlibpypy-c.*': No such file or directory`.

[fijal]

It's all fixed on trunk. Also please no longer use --gcrootfinder=shadowstack (as asmgcc has been fixed)

[unbit]

Latest binary nightly build works as-is without modifications: http://buildbot.pypy.org/nightly/trunk/pypy-c-jit-latest-linux64.tar.bz2 just use uwsgi-2.0 branch from github pointing --pypy-home to the pypy dir