search

avinassh / fast-sqlite3-inserts

Some bunch of test scripts to generate a SQLite DB with 1B rows in fastest possible way
MIT License
381 stars 38 forks source link

Rust reuse memory #10

Open pickfire opened 3 years ago

pickfire commented 3 years ago

Rather than creating the Vec in the inner loop, you can create it in the outer loop using with_capacity with capacity of batch size so it will not allocate in inner loop, then just reuse the Vec.

Once the execute is used, can just Vec clear to clear the items without deallocating memory. I wonder how much faster will it make.

dovahcrow commented 3 years ago

I tried this one and it gives me 9% speed up on the basic_batched benchmark.

avinassh commented 3 years ago

damn, thats amazing. Would you like to send a PR @pickfire @dovahcrow

darleybarreto commented 3 years ago

I would like to see that too. I tried to remove those ::new() from the loops, but I couldn't find a way to make the borrow checker happy.

pickfire commented 3 years ago

I just tried it, I seemed to have faced issues with borrow checker as well. I just use transmute to increase their lifetime and it worked. But yeah, people is not happy seeing unsafe. &dyn ToSql make it hard for the borrow checker I believe.

avinassh commented 3 years ago

@pickfire we can have another benchmark code just for unsafe and I don't mind unsafe code. may be we call it as batched_unsafe?

pickfire commented 3 years ago

@avinassh Without unsafe there is another set of optimization in place, it already have reduced allocations with that patch.