paradigmxyz / reth

Modular, contributor-friendly and blazing-fast implementation of the Ethereum protocol, in Rust
https://reth.rs/
Apache License 2.0
3.85k stars 1.1k forks source link

segment ring for fair + performant pruning on pruning interrupt #7343

Open emhane opened 6 months ago

emhane commented 6 months ago

Describe the feature

Segment pruning always starts with the same segment. Last segments in the iterator, risk not being pruned if the prune limit is small so it's often reached before pruning is done. We should implement a ring here: add a field to Pruner where we save the index in the chained segments iterator, of the last pruned segment. Upon calling prune_segments next time, the iterator should skip until the saved index + 1.

https://github.com/paradigmxyz/reth/blob/c04dbe6e9bd05be5da3a5d541adbf76166c14a08/crates/prune/src/pruner.rs#L141-L145

This will probably require modifying static_file_segments method, since it's important that no segments that need pruning are dropped when the iterator skips ahead to the last pruned segment.

https://github.com/paradigmxyz/reth/blob/c04dbe6e9bd05be5da3a5d541adbf76166c14a08/crates/prune/src/pruner.rs#L215-L243

Additional context

No response

github-actions[bot] commented 5 months ago

This issue is stale because it has been open for 21 days with no activity.

emhane commented 3 months ago

blocked by https://github.com/paradigmxyz/reth/issues/8746