search

Congyuwang / RocksDict

Python fast on-disk dictionary / RocksDB & SpeeDB Python binding
https://congyuwang.github.io/RocksDict/rocksdict.html
MIT License
173 stars 8 forks source link

Support for large data storage, and transformation #121

Closed umanniyaz closed 3 months ago

umanniyaz commented 4 months ago

Can it support storage for large data as a database.How transformations and CRUD operations impact its latency factor.

Congyuwang commented 3 months ago

I think it can support fairly large data size. It certainly is designed to store a large number of key-value pairs, especially when it cannot fit into the memory. For the specifics, refer to rocksdb, as this package is a thin wrapper around it.

umanniyaz commented 3 months ago

How to allocate maximum memory for storage of large key value pairs?

Congyuwang commented 3 months ago

What do you mean "allocate memory"?

Congyuwang commented 3 months ago

Did you meet some storage limitation or some other issue?

umanniyaz commented 3 months ago

When I am trying store huge data in key value pairs around 50 Million records my system crashes,is there a way to allocate some storage to Rdict to maximum limit

Congyuwang commented 3 months ago

What’s the error message?

Congyuwang commented 3 months ago

How many bytes are the keys and values on average, roughly? And are you sure that you have enough free disk space too?