Trigger level db compactRange

dapplion commented 2 years ago

From https://github.com/ChainSafe/lodestar/pull/4508#issuecomment-1239091324

Due to level-db internals, after a significant amount of data put or del it's necessary to compact that range to realize the size reduction. After an offline discussion with other teams:

Should compact all archived states range after the prune of this PR
Should compact after completing sync
Should consider compacting the DB at most once per 7 days

Given we have not done this before, extensive testing is necessary to:

Benchmark CPU and time cost of compactRange on different key ranges, plus how much size is free'ed on average situtations
Understand the impact on performance on the beacon node if any. While NodeJS is mostly single thread, I would expect this ops to happen in the C++ worker thread and not mess with validator performance

dapplion commented 2 years ago

I've added a new option on a branch https://github.com/ChainSafe/lodestar/compare/dapplion/bump-level...dapplion/test-compact which triggers compact of the entire db on start up with --compactDbOnStartup. I've deployed this branch in all feat2 hosts as of 11:20 CEST Sept 7th

Results for feat2-sm1v-ctvpss

node_filesystem_avail_bytes changed from 68GB -> 121GB: Total freed 53GB
Total time 13 min

Sep-07 11:27:00.863[]                 info: db compactRange on byte 0/256 allForks_stateArchive...
Sep-07 11:33:00.127[]                 info: db compactRange on byte 1/256 allForks_block...
Sep-07 11:33:02.192[]                 info: db compactRange on byte 2/256 allForks_blockArchive...
Sep-07 11:40:06.486[]                 info: db compactRange on byte 3/256 index_blockArchiveParentRootIndex...
Sep-07 11:40:11.251[]                 info: db compactRange on byte 4/256 index_blockArchiveRootIndex...
Sep-07 11:40:16.204[]                 info: db compactRange on byte 6/256 index_mainChain...
Sep-07 11:40:16.239[]                 info: db compactRange on byte 7/256 index_chainInfo...
Sep-07 11:40:16.273[]                 info: db compactRange on byte 8/256 phase0_eth1Data...
Sep-07 11:40:16.792[]                 info: db compactRange on byte 9/256 index_depositDataRoot...
Sep-07 11:40:16.897[]                 info: db compactRange on byte 12/256 phase0_depositData...
Sep-07 11:40:16.938[]                 info: db compactRange on byte 13/256 phase0_exit...
Sep-07 11:40:16.972[]                 info: db compactRange on byte 14/256 phase0_proposerSlashing...
Sep-07 11:40:17.009[]                 info: db compactRange on byte 15/256 phase0_attesterSlashing...
Sep-07 11:40:17.167[]                 info: db compactRange on byte 19/256 phase0_depositEvent...
Sep-07 11:40:17.593[]                 info: db compactRange on byte 20/256 phase0_slashingProtectionBlockBySlot...
Sep-07 11:40:17.627[]                 info: db compactRange on byte 21/256 phase0_slashingProtectionAttestationByTarget...
Sep-07 11:40:17.657[]                 info: db compactRange on byte 22/256 phase0_slashingProtectionAttestationLowerBound...
Sep-07 11:40:17.696[]                 info: db compactRange on byte 23/256 index_slashingProtectionMinSpanDistance...
Sep-07 11:40:17.735[]                 info: db compactRange on byte 24/256 index_slashingProtectionMaxSpanDistance...
Sep-07 11:40:17.831[]                 info: db compactRange on byte 26/256 index_stateArchiveRootIndex...
Sep-07 11:40:17.966[]                 info: db compactRange on byte 30/256 phase0_preGenesisState...
Sep-07 11:40:17.998[]                 info: db compactRange on byte 31/256 phase0_preGenesisStateLastProcessedBlock...
Sep-07 11:40:18.416[]                 info: db compactRange on byte 41/256 validator_metaData...
Sep-07 11:40:18.456[]                 info: db compactRange on byte 42/256 backfilled_ranges...
Sep-07 11:40:18.793[]                 info: db compactRange on byte 51/256 lightClient_syncCommitteeWitness...
Sep-07 11:40:19.391[]                 info: db compactRange on byte 52/256 lightClient_syncCommittee...
Sep-07 11:40:19.419[]                 info: db compactRange on byte 53/256 lightClient_checkpointHeader...
Sep-07 11:40:19.813[]                 info: db compactRange on byte 54/256 lightClient_bestPartialLightClientUpdate...
Sep-07 11:40:19.844[]                 info: db compactRange on byte 55/256 -...
...
Sep-07 11:40:27.167[]                 info: db compactRange on byte 254/256 -...
Sep-07 11:40:27.201[]                 info: db compactRange completed

Results for feat2-novc-ctvpss

node_filesystem_avail_bytes changed from 68.5GB -> 120GB: Total freed 51.5GB
Total time 12 min

devops@feat2-novc-ctvpss:~$ grep compactRange beacon/beacon-2022-09-07.log
Sep-07 11:27:00.795[]                 info: db compactRange on byte 0/256 allForks_stateArchive...
Sep-07 11:33:10.167[]                 info: db compactRange on byte 1/256 allForks_block...
Sep-07 11:33:11.898[]                 info: db compactRange on byte 2/256 allForks_blockArchive...
Sep-07 11:38:18.848[]                 info: db compactRange on byte 3/256 index_blockArchiveParentRootIndex...
Sep-07 11:38:23.226[]                 info: db compactRange on byte 4/256 index_blockArchiveRootIndex...
Sep-07 11:38:26.930[]                 info: db compactRange on byte 6/256 index_mainChain...
Sep-07 11:38:26.966[]                 info: db compactRange on byte 7/256 index_chainInfo...
Sep-07 11:38:27.005[]                 info: db compactRange on byte 8/256 phase0_eth1Data...
Sep-07 11:38:27.822[]                 info: db compactRange on byte 9/256 index_depositDataRoot...
Sep-07 11:38:27.906[]                 info: db compactRange on byte 12/256 phase0_depositData...
Sep-07 11:38:27.939[]                 info: db compactRange on byte 13/256 phase0_exit...
Sep-07 11:38:27.966[]                 info: db compactRange on byte 14/256 phase0_proposerSlashing...
Sep-07 11:38:27.993[]                 info: db compactRange on byte 15/256 phase0_attesterSlashing...
Sep-07 11:38:28.105[]                 info: db compactRange on byte 19/256 phase0_depositEvent...
Sep-07 11:38:28.240[]                 info: db compactRange on byte 20/256 phase0_slashingProtectionBlockBySlot...
Sep-07 11:38:28.267[]                 info: db compactRange on byte 21/256 phase0_slashingProtectionAttestationByTarget...
Sep-07 11:38:28.298[]                 info: db compactRange on byte 22/256 phase0_slashingProtectionAttestationLowerBound...
Sep-07 11:38:28.327[]                 info: db compactRange on byte 23/256 index_slashingProtectionMinSpanDistance...
Sep-07 11:38:28.354[]                 info: db compactRange on byte 24/256 index_slashingProtectionMaxSpanDistance...
Sep-07 11:38:28.419[]                 info: db compactRange on byte 26/256 index_stateArchiveRootIndex...
Sep-07 11:38:28.532[]                 info: db compactRange on byte 30/256 phase0_preGenesisState...
Sep-07 11:38:28.560[]                 info: db compactRange on byte 31/256 phase0_preGenesisStateLastProcessedBlock...
Sep-07 11:38:28.866[]                 info: db compactRange on byte 41/256 validator_metaData...
Sep-07 11:38:28.891[]                 info: db compactRange on byte 42/256 backfilled_ranges...
Sep-07 11:38:29.192[]                 info: db compactRange on byte 51/256 lightClient_syncCommitteeWitness...
Sep-07 11:38:29.825[]                 info: db compactRange on byte 52/256 lightClient_syncCommittee...
Sep-07 11:38:30.121[]                 info: db compactRange on byte 53/256 lightClient_checkpointHeader...
Sep-07 11:38:30.563[]                 info: db compactRange on byte 54/256 lightClient_bestPartialLightClientUpdate...
Sep-07 11:38:30.589[]                 info: db compactRange on byte 55/256 -...
...
Sep-07 11:38:37.320[]                 info: db compactRange on byte 254/256 -...
Sep-07 11:38:37.351[]                 info: db compactRange completed

Results for feat2-novc-ctvpss

Note this instance did not prune additional states beforehand, the two above prune states but did not compact

node_filesystem_avail_bytes changed from 128GB -> 128GB: Total freed 0GB
Total time 90 min

devops@feat2-md16-ctvpsm:~$ grep compactRange beacon/beacon-2022-09-07.log 
Sep-07 11:27:07.681[]                 info: db compactRange on byte 0/256 allForks_stateArchive...
Sep-07 12:26:58.457[]                 info: db compactRange on byte 0/256 allForks_stateArchive...
Sep-07 12:26:58.519[]                 info: db compactRange on byte 1/256 allForks_block...
Sep-07 12:27:00.476[]                 info: db compactRange on byte 2/256 allForks_blockArchive...
Sep-07 12:56:07.042[]                 info: db compactRange on byte 3/256 index_blockArchiveParentRootIndex...
Sep-07 12:57:24.598[]                 info: db compactRange completed

tmm360 commented 1 year ago

Any news on this? I'm running lodestar on gnosis, and periodically I have to delete db and redo sync because it is increasing by ~10GB per month.

dapplion commented 1 year ago

Any news on this? I'm running lodestar on gnosis, and periodically I have to delete db and redo sync because it is increasing by ~10GB per month.

This proposal was started to clean-up the consequences of an old bug where Lodestar persisted states too frequently. If you have re-synced Lodestar recently this proposal won't affect much. To reduce disk space we are working on this optimization which should reduce disk requirements significantly. I have to check Gnosis specific numbers, but for Ethereum it could be save 10-30 GB / month

https://github.com/ChainSafe/lodestar/issues/5671

ChainSafe / lodestar

Trigger level db compactRange #4515