Congyuwang / RocksDict

Python fast on-disk dictionary / RocksDB & SpeeDB Python binding
https://congyuwang.github.io/RocksDict/rocksdict.html
MIT License
176 stars 8 forks source link

does RocksDict compatible with RocksDB-Python? #61

Closed zjuzhfbloodz closed 1 year ago

zjuzhfbloodz commented 1 year ago

Here is my code, I am trying to save some sst files and read with rocksdb in python:

from rocksdict import Rdict, Options, SstFileWriter
import random
import rocksdb

# generate some rand bytes
rand_bytes1 = [random.randbytes(200) for _ in range(100000)]
rand_bytes1.sort()

# write to file1.sst
writer = SstFileWriter(options=Options(raw_mode=True))
writer.open("file1.sst")
for k, v in zip(rand_bytes1, rand_bytes1):
    writer[k] = v

writer.finish()

# bulid Rdict
d = Rdict("tmp", options=Options(raw_mode=True))
d.ingest_external_file(["file1.sst"])
d.close()

# rocksdb read
db = rocksdb.DB("tmp", read_only=True, opts=rocksdb.Options())

And I am getting the error as below:

---------------------------------------------------------------------------
Corruption                                Traceback (most recent call last)
remote%2B10.97.150.93/opt/huawei/data1/z00811450/code/test.ipynb#Y211sdnNjb2RlLXJlbW90ZQ%3D%3D?line=22) db = rocksdb.DB("tmp", read_only=True, opts=rocksdb.Options())

File /opt/huawei/wisebrain/anaconda3/lib/python3.9/site-packages/rocksdb/_rocksdb.pyx:1636, in rocksdb._rocksdb.DB.__cinit__()

File /opt/huawei/wisebrain/anaconda3/lib/python3.9/site-packages/rocksdb/_rocksdb.pyx:76, in rocksdb._rocksdb.check_status()

Corruption: b'Corruption: unknown checksum type 4 from footer of tmp/000019.sst, while checking block at offset 22672231 size 35'

Does this mean that rocksdict is not compatible with rocksdb in python? Looking forward to anyone's help!

Congyuwang commented 1 year ago

rocksdict uses rocksdb 8.0 for the moment. So it maybe incompatible to rocksdb 6 which is used by python-rocksdb

Congyuwang commented 1 year ago

But it should be compatible with other rocksdb clients, e.g., java, c++, rust