missing snappy support in linux wheels

[metachris]

We're using plyvel v0.9 for neo-python and run the tests with downloaded fixtures (leveldb snapshots). Now I tried upgrading to plyvel v1.0.2, and there are now some errors that say plyvel._plyvel.CorruptionError: b'Corruption: corrupted compressed block contents': https://travis-ci.org/CityOfZion/neo-python/builds/329522567

Traceback (most recent call last):
File "/home/travis/build/CityOfZion/neo-python/neo/Utils/WalletFixtureTestCase.py", line 59, in setUpClass
    super(WalletFixtureTestCase, cls).setUpClass()
File "/home/travis/build/CityOfZion/neo-python/neo/Utils/BlockchainFixtureTestCase.py", line 51, in setUpClass
    cls._blockchain = TestLevelDBBlockchain(path=cls.leveldb_testpath())
File "/home/travis/build/CityOfZion/neo-python/neo/Implementations/Blockchains/LevelDB/LevelDBBlockchain.py", line 154, in __init__
    for key, value in self._db.iterator(prefix=DBPrefix.DATA_Block):
File "plyvel/_plyvel.pyx", line 829, in plyvel._plyvel.Iterator.__next__
File "plyvel/_plyvel.pyx", line 874, in plyvel._plyvel.Iterator.real_next
File "plyvel/_plyvel.pyx", line 91, in plyvel._plyvel.raise_for_status
plyvel._plyvel.CorruptionError: b'Corruption: corrupted compressed block contents'

[wbolster]

hmmm. weird.

plyvel has almost identical code between 0.9 and 1.x except for a) one extra option, for which the leveldb minimum version was bumped, and b) added byte-like input flexibility.

see https://plyvel.readthedocs.io/en/latest/news.html#plyvel-1-0-0

diff is at https://github.com/wbolster/plyvel/compare/0.9...1.0.0

(most of it are docs)

[wbolster]

see also #8 btw (rather old)

[metachris]

Narrowing it down, I'm having a database that was created with plyvel 0.9, and getting an error when trying to read from it with plyvel 1.0.3:

    version = self._db.get(DBPrefix.SYS_Version)
File "plyvel/_plyvel.pyx", line 310, in plyvel._plyvel.DB.get
File "plyvel/_plyvel.pyx", line 111, in plyvel._plyvel.db_get
File "plyvel/_plyvel.pyx", line 91, in plyvel._plyvel.raise_for_status
plyvel._plyvel.CorruptionError: b'Corruption: corrupted compressed block contents'

We do store some bytes and binary content.

[metachris]

Found the issue -- databases created on OSX cannot be opened in Linux with plyvel > 0.9.

Steps to reproduce:

• Create a DB on OSX with plyvel 1.0.3, copy it to Linux (eg using Docker)
• In Linux you can open this DB using plyvel 0.9
• In Linux, using plyvel >0.9 you cannot open the DB anymore (plyvel._plyvel.CorruptionError: b'Corruption: corrupted compressed block contents')

[wbolster]

any chance you can share a minimal tarfile with a fresh db created on osx with one db.put(foo, bar) key pair in it? i don't have a functional osx dev machine currently.

[metachris]

Interesting. If I prepare a very simple db like this it works on Linux too. Here is a more complex one that breaks plyvel >0.9 on Linux: leveldb-broken-in-linux.tar.gz (1.5mb, taken after syncing a few hundred NEO blocks).

We are using it with neo-python. A really large database we're using for running the tests and which can't be loaded with plyvel 1.0.3 can be found here: https://s3.us-east-2.amazonaws.com/cityofzion/fixtures/fixtures_v4.tar.gz (1 GB)

[wbolster]

hmmm. this might have to do with apparently missing snappy support in the linux wheels.

without local compilation (wheels):

$ mktmpenv -ppython3
$ pip install plyvel  # uses wheels
$ cdsitepackages 
$ ldd plyvel/_plyvel.cpython-36m-x86_64-linux-gnu.so 
    linux-vdso.so.1 =>  (0x00007ffd50ff2000)
    libleveldb-f4f9cef4.so.1.20 => /home/wbolster/.virtualenvs/tmp-f9276a9099c6299c/lib/python3.6/site-packages/plyvel/.libs/libleveldb-f4f9cef4.so.1.20 (0x00007fa810fa7000)
    libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007fa810c21000)
    libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fa8108cb000)
    libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007fa8106b4000)
    libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fa810495000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fa8100b5000)
    /lib64/ld-linux-x86-64.so.2 (0x00007fa811443000)

with local compilation (no wheels, leveldb and snappy dev headers installed):

$ mktmpenv -ppython3
$ pip install --no-binary :all: plyvel  # does not use wheels
$ cdsitepackages 
$ ldd plyvel/_plyvel.cpython-36m-x86_64-linux-gnu.so 
    linux-vdso.so.1 =>  (0x00007fffa7541000)
    libleveldb.so.1 => /usr/lib/x86_64-linux-gnu/libleveldb.so.1 (0x00007fb60c83c000)
    libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007fb60c4b6000)
    libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007fb60c29f000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fb60bebf000)
    libsnappy.so.1 => /usr/lib/x86_64-linux-gnu/libsnappy.so.1 (0x00007fb60bcb7000)
    libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fb60b961000)
    libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fb60b742000)
    /lib64/ld-linux-x86-64.so.2 (0x00007fb60ccd4000)

[wbolster]

i just pushed 6fc14cb to enable building docker wheels with snappy support:

6fc14cb build manylinux1 wheels with libsnappy installed

contrary to the 1.0.3 wheel which was not linked against snappy, the resulting 1.0.4 wheel is linked against snappy:

$ pip install /home/wbolster/Projects/wbolster/plyvel/dist/plyvel-1.0.4-cp36-cp36m-manylinux1_x86_64.whl
Processing /home/wbolster/Projects/wbolster/plyvel/dist/plyvel-1.0.4-cp36-cp36m-manylinux1_x86_64.whl
Installing collected packages: plyvel
  Found existing installation: plyvel 1.0.3
    Uninstalling plyvel-1.0.3:
      Successfully uninstalled plyvel-1.0.3
Successfully installed plyvel-1.0.4

$ ldd plyvel/_plyvel.cpython-36m-x86_64-linux-gnu.so 
    linux-vdso.so.1 =>  (0x00007ffe859d3000)
    libleveldb-8dd97d89.so.1.20 => /home/wbolster/.virtualenvs/tmp-4512d8c259765459/lib/python3.6/site-packages/plyvel/.libs/libleveldb-8dd97d89.so.1.20 (0x00007fea62142000)
    libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007fea61dbc000)
    libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fea61a66000)
    libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007fea6184f000)
    libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fea61630000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fea61250000)
    libsnappy-d12e2ecd.so.1.3.0 => /home/wbolster/.virtualenvs/tmp-4512d8c259765459/lib/python3.6/site-packages/plyvel/.libs/libsnappy-d12e2ecd.so.1.3.0 (0x00007fea61047000)
    /lib64/ld-linux-x86-64.so.2 (0x00007fea625e7000)

i have just pushed 1.0.4 to pypi:

https://plyvel.readthedocs.io/en/latest/news.html#plyvel-1-0-4

please give it a try!

[metachris]

Awesome, it works!

[wbolster]

the reason it may have worked for a very simple db with only one key/value pair is that leveldb has not yet compacted it into a sst, and the data is only in the log file (wal-like) with completed writes that are pending compaction.

not entirely sure, i am a bit rusty on the details since i recently dusted off this project after 3 years. ;)

[metachris]

You are a great maintainer <3

[wbolster]

💥 thanks for confirming! 💟