avinassh / fast-sqlite3-inserts

Some bunch of test scripts to generate a SQLite DB with 1B rows in fastest possible way
MIT License
381 stars 38 forks source link

Try reusing memory #15

Open pickfire opened 3 years ago

pickfire commented 3 years ago

Fix #10

Unsafe is probably not nice but I am not sure how to satisfy borrow checker there. I tried using mem::swap and related functions but it still seemed like the borrow checker does not know that the reference is not longer valid when vec.clear().

29s -> 27s

avinassh commented 3 years ago

Is it possible to avoid unsafe in this?

darleybarreto commented 3 years ago

Is it possible to avoid unsafe in this?

I can't think of anything to help avoid these unsafe here, but I'm not a Rust expert, so...

darleybarreto commented 3 years ago

One thing I tried to do was using named_params!,

for batch in values.iter() {
    let (a,b,c) = batch;
    row_values.push(named_params!{":area": a, ":age": b, ":active": c});
}

but Params is not implemented for &[&[(&str, &dyn ToSql)]], which is the slice from the row_values.

pickfire commented 3 years ago

I switched to use https://docs.rs/rsor/0.1.2/rsor/ rsor library as suggested by @upsuper (he suggested a related library with runtime checks). std does not provide these tools and it only have compile-time cost.

avinassh commented 3 years ago

I ran it and I did not see much difference:

./bench.sh
Sun Aug  8 14:06:19 IST 2021 [RUST] basic_batched.rs (100_000_000) inserts

real    0m30.255s
user    0m27.745s
sys 0m2.247s
pickfire commented 3 years ago

What's the difference? 1s?

avinassh commented 3 years ago

@pickfire yup, seems so.

pickfire commented 3 years ago

Yeah, but it's consistent 1s and reduced memory allocation, I am not sure how good the effect will be on other parts.