PyPy support

[darkjh]

I've just tried to install plyvel in pypy but got the following error:

Downloading/unpacking plyvel
  http://206524166f7446e3946dfb16b510c1e6:f826619838ff45aca13e68df173a622d@pypi.botify.com/simple/plyvel/ uses an insecure transport scheme (http). Consider using https if 206524166f7446e3946dfb16b510c1e6:f826619838ff45aca13e68df173a622d@pypi.botify.com has it available
  Downloading plyvel-0.9.tar.gz (121kB): 121kB downloaded
  Running setup.py (path:/home/darkjh/projects/python/pypy_env/build/plyvel/setup.py) egg_info for package plyvel

    warning: no files found matching '*' under directory 'doc/build/html'
    warning: no files found matching 'plyvel/*.pxi'
Installing collected packages: plyvel
  Running setup.py install for plyvel
    building 'plyvel._plyvel' extension
    cc -O2 -fPIC -Wimplicit -I/home/darkjh/projects/python/pypy_env/include -c plyvel/_plyvel.cpp -o build/temp.linux-x86_64-2.7/plyvel/_plyvel.o -Wall -g
    cc1plus: warning: command line option ‘-Wimplicit’ is valid for C/ObjC but not for C++
    plyvel/_plyvel.cpp: In function ‘int __Pyx_GetItemInt_ByteArray_Fast(PyObject*, Py_ssize_t, int, int)’:
    plyvel/_plyvel.cpp:18127:45: error: ‘PyByteArray_GET_SIZE’ was not declared in this scope
             length = PyByteArray_GET_SIZE(string);
                                                 ^
    plyvel/_plyvel.cpp:18130:65: error: ‘PyByteArray_AS_STRING’ was not declared in this scope
                 return (unsigned char) (PyByteArray_AS_STRING(string)[i]);
                                                                     ^
    plyvel/_plyvel.cpp:18136:61: error: ‘PyByteArray_AS_STRING’ was not declared in this scope
             return (unsigned char) (PyByteArray_AS_STRING(string)[i]);
                                                                 ^
    plyvel/_plyvel.cpp: In function ‘int __Pyx_SetItemInt_ByteArray_Fast(PyObject*, Py_ssize_t, unsigned char, int, int)’:
    plyvel/_plyvel.cpp:18144:45: error: ‘PyByteArray_GET_SIZE’ was not declared in this scope
             length = PyByteArray_GET_SIZE(string);
                                                 ^
    plyvel/_plyvel.cpp:18147:41: error: ‘PyByteArray_AS_STRING’ was not declared in this scope
                 PyByteArray_AS_STRING(string)[i] = (char) v;
                                             ^
    plyvel/_plyvel.cpp:18154:37: error: ‘PyByteArray_AS_STRING’ was not declared in this scope
             PyByteArray_AS_STRING(string)[i] = (char) v;
                                         ^
    plyvel/_plyvel.cpp: In function ‘int __Pyx_GetItemInt_ByteArray_Fast(PyObject*, Py_ssize_t, int, int)’:
    plyvel/_plyvel.cpp:18138:1: warning: control reaches end of non-void function [-Wreturn-type]
     }
     ^
    error: command 'cc' failed with exit status 1
    Complete output from command /home/darkjh/projects/python/pypy_env/bin/pypy -c "import setuptools, tokenize;__file__='/home/darkjh/projects/python/pypy_env/build/plyvel/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-iLSs8H-record/install-record.txt --single-version-externally-managed --compile --install-headers /home/darkjh/projects/python/pypy_env/include/site/python2.7:
    running install

running build

running build_py

creating build

creating build/lib.linux-x86_64-2.7

creating build/lib.linux-x86_64-2.7/plyvel

copying plyvel/__init__.py -> build/lib.linux-x86_64-2.7/plyvel

copying plyvel/_version.py -> build/lib.linux-x86_64-2.7/plyvel

running build_ext

building 'plyvel._plyvel' extension

creating build/temp.linux-x86_64-2.7

creating build/temp.linux-x86_64-2.7/plyvel

cc -O2 -fPIC -Wimplicit -I/home/darkjh/projects/python/pypy_env/include -c plyvel/_plyvel.cpp -o build/temp.linux-x86_64-2.7/plyvel/_plyvel.o -Wall -g

cc1plus: warning: command line option ‘-Wimplicit’ is valid for C/ObjC but not for C++

plyvel/_plyvel.cpp: In function ‘int __Pyx_GetItemInt_ByteArray_Fast(PyObject*, Py_ssize_t, int, int)’:

plyvel/_plyvel.cpp:18127:45: error: ‘PyByteArray_GET_SIZE’ was not declared in this scope

         length = PyByteArray_GET_SIZE(string);

                                             ^

plyvel/_plyvel.cpp:18130:65: error: ‘PyByteArray_AS_STRING’ was not declared in this scope

             return (unsigned char) (PyByteArray_AS_STRING(string)[i]);

                                                                 ^

plyvel/_plyvel.cpp:18136:61: error: ‘PyByteArray_AS_STRING’ was not declared in this scope

         return (unsigned char) (PyByteArray_AS_STRING(string)[i]);

                                                             ^

plyvel/_plyvel.cpp: In function ‘int __Pyx_SetItemInt_ByteArray_Fast(PyObject*, Py_ssize_t, unsigned char, int, int)’:

plyvel/_plyvel.cpp:18144:45: error: ‘PyByteArray_GET_SIZE’ was not declared in this scope

         length = PyByteArray_GET_SIZE(string);

                                             ^

plyvel/_plyvel.cpp:18147:41: error: ‘PyByteArray_AS_STRING’ was not declared in this scope

             PyByteArray_AS_STRING(string)[i] = (char) v;

                                         ^

plyvel/_plyvel.cpp:18154:37: error: ‘PyByteArray_AS_STRING’ was not declared in this scope

         PyByteArray_AS_STRING(string)[i] = (char) v;

                                     ^

plyvel/_plyvel.cpp: In function ‘int __Pyx_GetItemInt_ByteArray_Fast(PyObject*, Py_ssize_t, int, int)’:

plyvel/_plyvel.cpp:18138:1: warning: control reaches end of non-void function [-Wreturn-type]

 }

 ^

error: command 'cc' failed with exit status 1

----------------------------------------
Cleaning up...
Command /home/darkjh/projects/python/pypy_env/bin/pypy -c "import setuptools, tokenize;__file__='/home/darkjh/projects/python/pypy_env/build/plyvel/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-iLSs8H-record/install-record.txt --single-version-externally-managed --compile --install-headers /home/darkjh/projects/python/pypy_env/include/site/python2.7 failed with error code 1 in /home/darkjh/projects/python/pypy_env/build/plyvel
Traceback (most recent call last):
  File "app_main.py", line 75, in run_toplevel
  File "/home/darkjh/projects/python/pypy_env/bin/pip", line 11, in <module>
    sys.exit(main())
  File "/home/darkjh/projects/python/pypy_env/site-packages/pip/__init__.py", line 185, in main
    return command.main(cmd_args)
  File "/home/darkjh/projects/python/pypy_env/site-packages/pip/basecommand.py", line 161, in main
    text = '\n'.join(complete_log)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 42: ordinal not in range(128)

I know that plyvel does not target Pypy but just want to know if it's possible to make it work with reasonable effort? It seems Cython has some support to generate Pypy compat code.

[wbolster]

At some point I had pretty much all tests passing on Pypy (except for threading stuff if I remember correctly).

The Pypy compat code generated by Cython seems to lack PyByteArray_AS_STRING...

[darkjh]

It seems that pypy does not expose it https://bitbucket.org/pypy/pypy/issue/1692/expose-pybytearray_as_string-in-cpyext

Is there a workaround?

[wbolster]

A bytearray is only used once in Plyvel: https://github.com/wbolster/plyvel/blob/master/plyvel/_plyvel.pyx#L124

Actually it's only used for compatibility with Python 2 and Python 3; maybe it can be adapted to have an alternative for Pypy?

[darkjh]

I found that I can compile Cython codes (Make all) and install them under pypy (python setup.py develop under pypy virtualenv) and it seems working (batch write and iteration). But pip install does not work.

[wbolster]

Perhaps the Cython versions are different? Can you look at the versions?

[darkjh]

No I've tried install a Cython 0.19.2 in a pypy virtualenv then pip install plyvel but no success. Maybe I didn't use all features (juste writing and iterating) so it seems to work. (say I don't touch the part of bytearray).

[wbolster]

The error you show above is a compilation error. If the compilation fails, nothing works and the import will fail. So it's not possible it's partly working...

[darkjh]

@wbolster I just return to this issue (sry for delay). Can I just workaround the problem by replacing the bytearray by other things that pypy supports? Currently we are only on py2.

[darkjh]

If in this line: https://github.com/wbolster/plyvel/blob/master/plyvel/_plyvel.pyx#L130 I just assign b = s, will that work for py2 under pypy?

[wbolster]

No that won't work. The whole thing is written specifically because it is different in py2/py3, and bytearray makes it work on both. I think I can change it to not use bytearray if that would help for PyPI.

[bauerj]

Being able to run plyvel under PyPy would be a good thing for electrum-server. :+1:

[heikoheiko]

Being able to run plyvel under PyPy would be a good thing for Ethereum too. :+1:

[wbolster]

fwiw: unit tests pass on the latest pypy3.5

[wbolster]

closing this since it seems to work on recent pypy (tests pass)