Document leveldb requirements

[muelli]

I've just tried to install pyvel on a Debian Stretch:

$ lsb_release -a
No LSB modules are available.
Distributor ID: Debian
Description:    Debian GNU/Linux 9.3 (stretch)
Release:    9.3
Codename:   stretch

Installation fails:

pip install plyvel 
Collecting plyvel
  Using cached plyvel-1.0.0.tar.gz
Building wheels for collected packages: plyvel
  Running setup.py bdist_wheel for plyvel ... error
  Complete output from command /tmp/pscore-test/bin/python2 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-ZvXyeS/plyvel/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /tmp/tmpW5L2vtpip-wheel- --python-tag cp27:
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build/lib.linux-x86_64-2.7
  creating build/lib.linux-x86_64-2.7/plyvel
  copying plyvel/__init__.py -> build/lib.linux-x86_64-2.7/plyvel
  copying plyvel/_version.py -> build/lib.linux-x86_64-2.7/plyvel
  running build_ext
  building 'plyvel._plyvel' extension
  creating build/temp.linux-x86_64-2.7
  creating build/temp.linux-x86_64-2.7/plyvel
  x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fdebug-prefix-map=/build/python2.7-2.7.13=. -fstack-protector-strong -Wformat -Werror=format-security -fPIC -I/usr/include/python2.7 -c plyvel/_plyvel.cpp -o build/temp.linux-x86_64-2.7/plyvel/_plyvel.o -Wall -g
  cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
  plyvel/_plyvel.cpp: In function ‘int __pyx_f_6plyvel_7_plyvel_parse_options(leveldb::Options*, bool, bool, PyObject*, PyObject*, PyObject*, PyObject*, PyObject*, PyObject*, PyObject*, PyObject*, int, PyObject*, PyObject*)’:
  plyvel/_plyvel.cpp:3341:22: error: ‘struct leveldb::Options’ has no member named ‘max_file_size’; did you mean ‘max_open_files’?
       __pyx_v_options->max_file_size = __pyx_t_4;
                        ^~~~~~~~~~~~~
  error: command 'x86_64-linux-gnu-gcc' failed with exit status 1

  ----------------------------------------
  Failed building wheel for plyvel
  Running setup.py clean for plyvel
Failed to build plyvel
Installing collected packages: plyvel
  Running setup.py install for plyvel ... error
    Complete output from command /tmp/pscore-test/bin/python2 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-ZvXyeS/plyvel/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-4O4Q5j-record/install-record.txt --single-version-externally-managed --compile --install-headers /tmp/pscore-test/include/site/python2.7/plyvel:
    running install
    running build
    running build_py
    creating build
    creating build/lib.linux-x86_64-2.7
    creating build/lib.linux-x86_64-2.7/plyvel
    copying plyvel/__init__.py -> build/lib.linux-x86_64-2.7/plyvel
    copying plyvel/_version.py -> build/lib.linux-x86_64-2.7/plyvel
    running build_ext
    building 'plyvel._plyvel' extension
    creating build/temp.linux-x86_64-2.7
    creating build/temp.linux-x86_64-2.7/plyvel
    x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fdebug-prefix-map=/build/python2.7-2.7.13=. -fstack-protector-strong -Wformat -Werror=format-security -fPIC -I/usr/include/python2.7 -c plyvel/_plyvel.cpp -o build/temp.linux-x86_64-2.7/plyvel/_plyvel.o -Wall -g
    cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
    plyvel/_plyvel.cpp: In function ‘int __pyx_f_6plyvel_7_plyvel_parse_options(leveldb::Options*, bool, bool, PyObject*, PyObject*, PyObject*, PyObject*, PyObject*, PyObject*, PyObject*, PyObject*, int, PyObject*, PyObject*)’:
    plyvel/_plyvel.cpp:3341:22: error: ‘struct leveldb::Options’ has no member named ‘max_file_size’; did you mean ‘max_open_files’?
         __pyx_v_options->max_file_size = __pyx_t_4;
                          ^~~~~~~~~~~~~
    error: command 'x86_64-linux-gnu-gcc' failed with exit status 1

    ----------------------------------------

My guess is that the Debian leveldb headers are too old:

$ apt-cache policy  libleveldb-dev
libleveldb-dev:
  Installed: 1.18-5
  Candidate: 1.18-5
  Version table:
 *** 1.18-5 500
        500 http://ftp.uni-bayreuth.de/debian stretch/main amd64 Packages
        100 /var/lib/dpkg/status
$

this suggests, that 1.20 is required:

https://github.com/wbolster/plyvel/blob/fa460e431982e94034fe226faef570ce498c89ac/travis.sh#L5

I first looked through the documentation to find a statement about the required version, but couldn't find any. My life would have been a tiny bit happier if the installation instructions provided a hint towards the required version. Maybe even via the travis.sh file. My life would be even more joyful if there was a documented way to get plyvel running on a current stable Debian.

[wbolster]

it's documented in the release notes at https://plyvel.readthedocs.io/en/latest/news.html

anyway i ran into the same issue :)

i am investigating building manylinux1 wheels with embedded leveldb for linux and publishing those on pypi.

[wbolster]

btw just use the previous version in the mean time

[muelli]

it's documented in the release notes at https://plyvel.readthedocs.io/en/latest/news.html

fair enough. I didn't see that. I consider it to be a bit weird though to have the legacy Ubuntu version information in the "Installation" section but without details reg. the required versions.

[wbolster]

pull requests welcome. side project, first release in over three years. :)

[wbolster]

i wrote in https://github.com/wbolster/plyvel/issues/62#issuecomment-355553085

i am investigating building manylinux1 wheels with embedded leveldb for linux and publishing those on pypi.

see #64 for initial work on this.

[wbolster]

plyvel 1.0.1 is on pypi, with binary linux packages! please give it a try!

https://plyvel.readthedocs.io/en/latest/news.html#plyvel-1-0-1

https://twitter.com/wbolster/status/949394305893924865

[wbolster]

https://plyvel.readthedocs.io/en/latest/installation.html

is now updated.

also, binary pre-built packages should help a lot.

[wbolster]

My life would be even more joyful if there was a documented way to get plyvel running on a current stable Debian.

please let me know whether my investment of spare time made your life even more joyful ;)

[muelli]

$ pip install --upgrade plyvel
Collecting plyvel
  Downloading plyvel-1.0.1-cp27-cp27mu-manylinux1_x86_64.whl (526kB)
    100% |████████████████████████████████| 532kB 1.2MB/s 
Installing collected packages: plyvel
  Found existing installation: plyvel 0.9
    Uninstalling plyvel-0.9:
      Successfully uninstalled plyvel-0.9
Successfully installed plyvel-1.0.1
$

sweet. I've just stood up and danced around my desk. With a festive hat. Thanks. Your next beer is on me.

[muelli]

now I've returned and my joyful times are over:

$ .pyenv/bin/python demo.py 
Traceback (most recent call last):
  File "demo.py", line 1, in <module>
    from automation import TaskManager, CommandSequence
  File "./tests/vendor/OpenWPM/automation/TaskManager.py", line 2, in <module>
    from DataAggregator import DataAggregator, LevelDBAggregator
  File "./tests/vendor/OpenWPM/automation/DataAggregator/LevelDBAggregator.py", line 3, in <module>
    import plyvel
  File "./tests/vendor/OpenWPM/.pyenv/local/lib/python2.7/site-packages/plyvel/__init__.py", line 6, in <module>
    from ._plyvel import (  # noqa
ImportError: ./tests/vendor/OpenWPM/.pyenv/local/lib/python2.7/site-packages/plyvel/_plyvel.so: undefined symbol: PyUnicodeUCS2_FromObject
$ ldd ./tests/vendor/OpenWPM/.pyenv/local/lib/python2.7/site-packages/plyvel/_plyvel.so
    linux-vdso.so.1 (0x00007fff08bc9000)
    libleveldb-f4f9cef4.so.1.20 => ./tests/vendor/OpenWPM/.pyenv/local/lib/python2.7/site-packages/plyvel/.libs/libleveldb-f4f9cef4.so.1.20 (0x00007f9cfb7f7000)
    libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f9cfb475000)
    libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f9cfb171000)
    libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f9cfaf5a000)
    libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f9cfad3d000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f9cfa99e000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f9cfbc92000)
$ 

But this report was not about binary compatibility but about the documentation. And that's been nicely fixed. So I'll continue my life with the 0.9 version for now.

[wbolster]

weird, i uploaded both m and mu builds for plyvel

• plyvel-1.0.1-cp27-cp27m-manylinux1_x86_64.whl
• plyvel-1.0.1-cp27-cp27mu-manylinux1_x86_64.whl

https://pypi.org/project/plyvel/#files

please open a new issue.

[wbolster]

looks similar to https://github.com/pantsbuild/pex/issues/398 btw

[wbolster]

this should be fixed for python2.7 in plyvel 1.0.2 which i just released