search

twmht / python-rocksdb

Python bindings for RocksDB
BSD 3-Clause "New" or "Revised" License
274 stars 89 forks source link

Compact Range in pybind11 branch #63

Closed oersted closed 5 years ago

oersted commented 5 years ago

My use case is the following: create a RocksDB index as a batch process, then let multiple processes open it as read-only and use the index repeatedly.

It is a rather basic use case, but I am currently unable to implement it in the latest python-rocksdb master release because there is no reliable way to close a DB (see #58) and not all data is written to disk when the index creation is done.

Following the advice from #58, I ported this section of my code to the pybind11 branch. Now, this is far from ideal, I assume pybind11 to be rather unstable and many features are missing, but at least DB closing works.

However, I would like the index to be fast for read-only reads. As far as I understand, compaction can't happen in read-only mode and closing a DB doesn't necessarily force compaction. So, if the DB is not compacted during creation, it will be relatively slow for reads. And since such few options are exposed in the API, I am also unable to make background compaction more aggressive.

So, in essence, I would be very grateful if compact_range were exposed to the API in pybind11.

Please let me know if you have any alternative suggestions that might help me in my use case.

twmht commented 5 years ago

So you need compact range API for doing this? https://github.com/twmht/python-rocksdb/blob/pybind11/tests/test_db.py#L384

oersted commented 5 years ago

Ah, good, thanks. It wasn't documented in the docs.

twmht commented 5 years ago

yup

it's not documented yet. I will update the document as soon as possible.

twmht commented 5 years ago

By the way, feel free to open any feature request on issue list since it's hard to export all of apis of rocksdb. But it's much more easily to export only the most used APIs.