In the WriteBatch struct, I changed the Vec<Box<[u8]>> fields to Vec<[u8, N]>, which saves on memory allocations when indexing blocks and putting the results of that in the db, before this pr that adds up to 1 per unique output address per block (funding), 1 per spent utxo per block (spending) and 1 per transaction (txid) (and a bit more for tip_row and header_rows).
~When retrieving from the db currently Boxes are still used, but this can be fixed by using rocksdb's raw iterators. (The optimizer might be able to even remove the Boxes now that they are directly dropped, but as with almost all optimizations, they are not guaranteed).~
I also managed to fix this, 800000+ less Boxes are now made when retrieving the headers on startup, and a lot too on electrum protocol requests (i think the amount here depends on the amount of blocks the address is used in, for history and balance at least)
And when loading headers, the iterator is not .collect().into_iter()'ed anymore (found that 2 times, so another 2 intermediate allocations saved).
I was experimenting with redb a bit, redb uses fixed size types, unlike rocksdb that only allows &[u8], this change would also make it easier to use redb as db with electrs in the future, I plan on making a pr for adding redb if I get it to work.
In the WriteBatch struct, I changed the
Vec<Box<[u8]>>fields toVec<[u8, N]>, which saves on memory allocations when indexing blocks and putting the results of that in the db, before this pr that adds up to 1 per unique output address per block (funding), 1 per spent utxo per block (spending) and 1 per transaction (txid) (and a bit more for tip_row and header_rows).~When retrieving from the db currently
Boxes are still used, but this can be fixed by using rocksdb's raw iterators. (The optimizer might be able to even remove theBoxes now that they are directly dropped, but as with almost all optimizations, they are not guaranteed).~ I also managed to fix this, 800000+ lessBoxes are now made when retrieving the headers on startup, and a lot too on electrum protocol requests (i think the amount here depends on the amount of blocks the address is used in, for history and balance at least)And when loading headers, the iterator is not
.collect().into_iter()'ed anymore (found that 2 times, so another 2 intermediate allocations saved).I was experimenting with redb a bit, redb uses fixed size types, unlike rocksdb that only allows
&[u8], this change would also make it easier to use redb as db with electrs in the future, I plan on making a pr for adding redb if I get it to work.