Open shamil-gadelshin opened 2 months ago
Tl;DR is basically the following:
1) Chain is at Genesis
2) We implemented downloading of block and its state
3) We need to find a way to insert this downloaded block with state (useally using normal ImportQueueService::import_blocks
such that we don't hit any errors and afterwards everything looks like a normal pruned node
@bkchr may I bother you to help with this a bit? Or maybe there is someone with good knowledge of this part of the codebase. I suspect this is probably not doable with nice public APIs right now, but maybe there is a small gap to bridge so we can get what we need here?
UPD: Think of it like warp sync, but without gap sync that follows.
What is the correct (easiest) way to sync the blockchain state to a specific block in Substrate?
We'd like to develop a way to fast-sync our Substrate-based blockchain similar to warp-sync or similar techniques. This is our current idea:
We developed a solution based on Substrate codebase.
SyncMode::LightState
to allow bypassing some checks.lock_import_and_run
andapply_block
.StateSync
syncing strategy to download and insert the state by utilizing a simplified form of thesyncing-engine
.block gap
data from the blockchain to skip downloading the previous history. b) Reinitialize the syncing engine and its strategies withSyncMode::Full
to download the full blocks. c) Update the known common block values with the target block number when nodes are synced themselves.The core of this functionality (2. inserting a target block and 3. obtaining the correct state) is rather straightforward and requires only exporting some Substrate API as public. However, additional "hacks" like "1. Use SyncMode::LightState" and "4. Set internal data structures" seem invasive. Actually, "4.b) Reinitialize the syncing engine to use
SyncMode::Full
" and "4.c) Update the known common block values" are redundant because SubstrateChainSync
strategy can auto-recover from these events but it produces error messages in the process - so it's UX improvement.1.Node configuration with
SyncMode::LightState
allows bypassing some checks - specifically the check forNonCanonicalOverlay
. The error seems to occur becauseSyncMode::Full
setscommit_state
variable totrue
when starting from genesis in contrast toLightState
andtry_commit_operation
operation goes to the branch where it triggerscanonicalize_block
which in turn goes toNonCanonicalOverlay
and triggers the finalInvalidBlockNumber
error.4.a) Block gap is set during the
try_commit_operation
when the blockchain detects the difference between the best block and the block we're trying to insert is greater than 1.So, how do we insert a specific block in the blockchain and download the state for it without triggering errors? How to disable gap sync download properly? Is there a simple way to achieve our goal without the described techniques?
We appreciate your comments.