search

solana-labs / solana

Web-Scale Blockchain for fast, secure, scalable, decentralized apps and marketplaces.
https://solanalabs.com
Apache License 2.0
13.19k stars 4.3k forks source link

Mitigate RocksDB IO spike when rpc-history is enabled #33961

Open yhchiang-sol opened 1 year ago

yhchiang-sol commented 1 year ago

Problem

When rpc-history is enabled on a validator, blockstore will create additional RocksDB column families. The keys of those rpc-related column families are pubkey-prefixed for faster prefix-scan operation. As a result, their underlying RocksDB SST files are not sorted by slot and thus their disk space reclaiming is less performant than other slot-based column families as our disk space reclaiming logic is slot-based.

This makes IO spikes when the periodic compaction starts.

Proposed Solution

The problem is fundamentally challenging as it's impossible to have one key format that satisfies both read and compaction as they require different key prefixes. However, the IO spike issue can be mitigated by using RocksDB's RateLimiter which will smooth out the IO spikes.

yhchiang-sol commented 1 year ago

Created https://github.com/facebook/rocksdb/pull/12058 to expose auto-tuned RateLimiter in RocksDB's C API.