search

Congyuwang / RocksDict

Python fast on-disk dictionary / RocksDB & SpeeDB Python binding
https://congyuwang.github.io/RocksDict/rocksdict.html
MIT License
173 stars 8 forks source link

Is there a fast way to get all keys available on the DB #133

Closed orena1 closed 2 months ago

orena1 commented 2 months ago

The only method that I say is: list(Rdict(f'...DBS/rocksdict/{date}',options=opt).keys()) but it takes a lot of time, from what I understand it iterate over all the data. Thanks,

Congyuwang commented 2 months ago

I think this already is the fastest way. Use raw mode too.

Congyuwang commented 2 months ago

All data have to be retrieved from the disk, since this is by design what rocksdb does. It assumes that there is difficulty storing everything inside main memory.

However, from what I can see, in your use case, there is no difficulty keeping all keys within main memory. So, what you can do is perhaps just keep all your keys in memory, and also perhaps separately deserialize all your keys in another format for fast batch retrieval.