Closed metachris closed 6 years ago
hmmm. weird.
plyvel has almost identical code between 0.9 and 1.x except for a) one extra option, for which the leveldb minimum version was bumped, and b) added byte-like input flexibility.
see https://plyvel.readthedocs.io/en/latest/news.html#plyvel-1-0-0
diff is at https://github.com/wbolster/plyvel/compare/0.9...1.0.0
(most of it are docs)
see also #8 btw (rather old)
Narrowing it down, I'm having a database that was created with plyvel 0.9, and getting an error when trying to read from it with plyvel 1.0.3:
version = self._db.get(DBPrefix.SYS_Version)
File "plyvel/_plyvel.pyx", line 310, in plyvel._plyvel.DB.get
File "plyvel/_plyvel.pyx", line 111, in plyvel._plyvel.db_get
File "plyvel/_plyvel.pyx", line 91, in plyvel._plyvel.raise_for_status
plyvel._plyvel.CorruptionError: b'Corruption: corrupted compressed block contents'
We do store some bytes and binary content.
Found the issue -- databases created on OSX cannot be opened in Linux with plyvel > 0.9.
Steps to reproduce:
plyvel._plyvel.CorruptionError: b'Corruption: corrupted compressed block contents'
)any chance you can share a minimal tarfile with a fresh db created on osx with one db.put(foo, bar) key pair in it? i don't have a functional osx dev machine currently.
Interesting. If I prepare a very simple db like this it works on Linux too. Here is a more complex one that breaks plyvel >0.9 on Linux: leveldb-broken-in-linux.tar.gz (1.5mb, taken after syncing a few hundred NEO blocks).
We are using it with neo-python. A really large database we're using for running the tests and which can't be loaded with plyvel 1.0.3 can be found here: https://s3.us-east-2.amazonaws.com/cityofzion/fixtures/fixtures_v4.tar.gz (1 GB)
hmmm. this might have to do with apparently missing snappy support in the linux wheels.
without local compilation (wheels):
$ mktmpenv -ppython3
$ pip install plyvel # uses wheels
$ cdsitepackages
$ ldd plyvel/_plyvel.cpython-36m-x86_64-linux-gnu.so
linux-vdso.so.1 => (0x00007ffd50ff2000)
libleveldb-f4f9cef4.so.1.20 => /home/wbolster/.virtualenvs/tmp-f9276a9099c6299c/lib/python3.6/site-packages/plyvel/.libs/libleveldb-f4f9cef4.so.1.20 (0x00007fa810fa7000)
libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007fa810c21000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fa8108cb000)
libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007fa8106b4000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fa810495000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fa8100b5000)
/lib64/ld-linux-x86-64.so.2 (0x00007fa811443000)
with local compilation (no wheels, leveldb and snappy dev headers installed):
$ mktmpenv -ppython3
$ pip install --no-binary :all: plyvel # does not use wheels
$ cdsitepackages
$ ldd plyvel/_plyvel.cpython-36m-x86_64-linux-gnu.so
linux-vdso.so.1 => (0x00007fffa7541000)
libleveldb.so.1 => /usr/lib/x86_64-linux-gnu/libleveldb.so.1 (0x00007fb60c83c000)
libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007fb60c4b6000)
libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007fb60c29f000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fb60bebf000)
libsnappy.so.1 => /usr/lib/x86_64-linux-gnu/libsnappy.so.1 (0x00007fb60bcb7000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fb60b961000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fb60b742000)
/lib64/ld-linux-x86-64.so.2 (0x00007fb60ccd4000)
i just pushed 6fc14cb to enable building docker wheels with snappy support:
6fc14cb build manylinux1 wheels with libsnappy installed
contrary to the 1.0.3 wheel which was not linked against snappy, the resulting 1.0.4 wheel is linked against snappy:
$ pip install /home/wbolster/Projects/wbolster/plyvel/dist/plyvel-1.0.4-cp36-cp36m-manylinux1_x86_64.whl
Processing /home/wbolster/Projects/wbolster/plyvel/dist/plyvel-1.0.4-cp36-cp36m-manylinux1_x86_64.whl
Installing collected packages: plyvel
Found existing installation: plyvel 1.0.3
Uninstalling plyvel-1.0.3:
Successfully uninstalled plyvel-1.0.3
Successfully installed plyvel-1.0.4
$ ldd plyvel/_plyvel.cpython-36m-x86_64-linux-gnu.so
linux-vdso.so.1 => (0x00007ffe859d3000)
libleveldb-8dd97d89.so.1.20 => /home/wbolster/.virtualenvs/tmp-4512d8c259765459/lib/python3.6/site-packages/plyvel/.libs/libleveldb-8dd97d89.so.1.20 (0x00007fea62142000)
libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007fea61dbc000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fea61a66000)
libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007fea6184f000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fea61630000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fea61250000)
libsnappy-d12e2ecd.so.1.3.0 => /home/wbolster/.virtualenvs/tmp-4512d8c259765459/lib/python3.6/site-packages/plyvel/.libs/libsnappy-d12e2ecd.so.1.3.0 (0x00007fea61047000)
/lib64/ld-linux-x86-64.so.2 (0x00007fea625e7000)
i have just pushed 1.0.4 to pypi:
https://plyvel.readthedocs.io/en/latest/news.html#plyvel-1-0-4
please give it a try!
Awesome, it works!
the reason it may have worked for a very simple db with only one key/value pair is that leveldb has not yet compacted it into a sst
, and the data is only in the log file (wal-like) with completed writes that are pending compaction.
not entirely sure, i am a bit rusty on the details since i recently dusted off this project after 3 years. ;)
You are a great maintainer <3
š„ thanks for confirming! š
We're using plyvel v0.9 for neo-python and run the tests with downloaded fixtures (leveldb snapshots). Now I tried upgrading to plyvel v1.0.2, and there are now some errors that say
plyvel._plyvel.CorruptionError: b'Corruption: corrupted compressed block contents'
: https://travis-ci.org/CityOfZion/neo-python/builds/329522567