stephan-hof / pyrocksdb

Python bindings for RocksDB
BSD 3-Clause "New" or "Revised" License
150 stars 169 forks source link

Optimization: Support batch size arguments for iterables #38

Open Downchuck opened 8 years ago

Downchuck commented 8 years ago

I'm going to take a peek at the performance on this -- if it's not worth it, I'll just close out the issue.

There may be overhead in the iterkeys/iteritems/itervalues calls when traversing a large set of data, which could be reduced by running a loop of iteration calls in the cython interface before returning results.

My goal is to show a significant increase in speed when traversing through more than a hundred thousand records. I'll just implement an iterbatch() method for the moment.