search

twmht / python-rocksdb

Python bindings for RocksDB
BSD 3-Clause "New" or "Revised" License
274 stars 89 forks source link

How to optimize the speed when I import a large data into pyrocksdb? #85

Open xiaolinpan opened 3 years ago

xiaolinpan commented 3 years ago

When I import a large data into rocksdb, the speed is very slow. How can I optimize the speed？thank you!

Jamiejoin commented 3 years ago

Replace the hard disk with an SSD, and provide your rocksdb status log, search for some optimization solutions about delayed write and write blocking，If you are importing existing data to rocksdb, you can try to create an sst file to import

ckingdev commented 3 years ago

If you are doing an initial bulk load, insert the data in order, sorted by key. Sorting the data up front makes for much less work when loading.

You can also create the SST files yourself- https://github.com/facebook/rocksdb/wiki/Creating-and-Ingesting-SST-files