paritytech / parity-bitcoin

The Parity Bitcoin client
GNU General Public License v3.0
730 stars 215 forks source link

In-memory storage for headers #285

Open svyatonik opened 7 years ago

svyatonik commented 7 years ago

from slack: guys, for getblock RPC method I need chainwork for the block. I have found two approaches to get this. First one is to store this with block in the db - thats how bitcoinj works: https://github.com/bitcoinj/bitcoinj/blob/a7cad0ede447d4bcba7dd55639df442a408df6fb/core/src/main/java/org/bitcoinj/core/StoredBlock.java#L50 the second one is to calculate it on-the-fly. For this I have to read all previous blocks => this is not very effective, but...bitcoind works this way && I suppose it actually loads all headers from db on start && calculates chainwork for each header. So it stores all headers in-memory => there's large performance boost for sync (& maybe for some parts of verification). But also larger memory consumption (which I suppose is ok if we're targeting miners): https://github.com/bitcoin/bitcoin/blob/master/src/validation.cpp#L3341 what do you think? which approach should we use? [11:56]
iirc it takes ~40Mb to hold all headers in-memory, so it's not that big. I feel that we should take this approach

So: 1) storage must be filled up on start 2) storage must be updated when block is accepted by the storage 3) storage should also store chainwork + be able to calculate height for each block_header (not just for main block headers)

@NikVolf we should discuss - how to implement it - as a cache-level in Store or as a separate entity somewhere in sync. I think that it's better to hide this behind db::Store, but that's just my opinion :)

debris commented 7 years ago

Last 2k inserted headers are now stored in memory https://github.com/paritytech/parity-bitcoin/blob/362d9bf2dc3474687f31852afd105b07499656f8/db/src/kv/cachedb.rs We can increase that number if needed