paradigmxyz / reth

Modular, contributor-friendly and blazing-fast implementation of the Ethereum protocol, in Rust
https://reth.rs/
Apache License 2.0
3.52k stars 906 forks source link

History pruning pre-computations in a read transaction #9181

Open shekhirin opened 1 week ago

shekhirin commented 1 week ago

Describe the feature

Ref https://github.com/paradigmxyz/reth/issues/8995

What if we do as much of accounts/storage history pruning work as possible in a read transaction and then open a write transaction only to seek to the entries that need modification/deletion.

What we can do here is history pruning lagged by one or several blocks:

  1. We start calculating the history indices that need pruning in a background read transaction after committing the block N
  2. Pruning starts after block N+1, and we use pre-computed indices from step 1. This will not prune data for block N+1, but it's not a big deal and we'll prune it later.

It would also be useful to collect Samply profiles for history pruning to see what takes the most time.

Additional context

No response

### Tasks
- [ ] https://github.com/paradigmxyz/reth/issues/9183
mattsse commented 6 days ago

all of this should be done concurrent to engine request processing with #8742 will let us do since this requires a write tx this must be coordinated, and handled XOR to db commits.

what is the expensive part during pruning, is it read related or write? imo we can like split this across txs because our assumptions here are:

shekhirin commented 4 days ago

Samply profile

image
emhane commented 2 days ago

blocked by https://github.com/paradigmxyz/reth/issues/8746