search

wbolster / plyvel

Plyvel, a fast and feature-rich Python interface to LevelDB
https://plyvel.readthedocs.io/
Other
530 stars 75 forks source link

write_batch writing slowing down #45

Closed edz504 closed 8 years ago

edz504 commented 8 years ago

Hi there,

I'm trying to open an existing LevelDB (A) with plyvel, create a new LevelDB (B), and write the first X values of A to B. I'm doing this in batch writes of 100, but for some reason, the first few batches write extremely quickly, and then as the process continues, the writing slows down tremendously. I'm talking from < 1 second for the first batch write to ~10 seconds for the 70th batch.

I'd like to know if this is expected (as B grows in size?), or if there's possibly a small mistake I'm making that's slowing the process down.

Also, if anyone has any suggestions for a better way to subset part of A into a new LevelDB, please let me know.

Thanks!

wbolster commented 8 years ago

any chance you can test without batching? and any chance you can check memory use while running (with batching)?

— wouter

sent from my phone. please forgive my tpyos.

On February 16, 2016 4:53:40 PM GMT+01:00, Eddie Zhou notifications@github.com wrote:

Hi there,

I'm trying to open an existing LevelDB (A) with plyvel, create a new LevelDB (B), and write the first X values of A to B. I'm doing this in batch writes of 100, but for some reason, the first few batches write extremely quickly, and then as the process continues, the writing slows down tremendously. I'm talking from < 1 second for the first batch write to ~10 seconds for the 70th batch.

I'd like to know if this is expected (as B grows in size?), or if there's possibly a small mistake I'm making that's slowing the process down.

Also, if anyone has any suggestions for a better way to subset part of A into a new LevelDB, please let me know.

Thanks!

Reply to this email directly or view it on GitHub: https://github.com/wbolster/plyvel/issues/45

edz504 commented 8 years ago

Hi -- figured out my problem, it wasn't plylevel related. Thanks, though!

wbolster commented 8 years ago

btw keep in mind that write batches are intended for atomicity, and will generally not make writes faster.