fix
commented
8 years ago Hi @karek314 thanks for the proposal this is great!
We have already a few fixes for the block sync, but far from enough.
So far the implementation is the following:
- new node is looking at the network for 100 random peers
- the peer query height on this peer and building an histogram to estimate network height (available at 0.4.0)
- the peer select a list of peers at the estimated network height using histogram cut (available at 0.4.0)
- the peer download batch of 1440 blocks for a random peer
- the random peer is choosen from the list 3. and change every batch of blocks (available in 0.4.0)
- a minor bottleneck is the download time. Being able to zip batches of blocks will surely improve it.
- the major time bottleneck is block processing (above all transactions processing). To give you an idea a block with 25 tx takes 1s in the current implementation.
bottom-line: decentralised block chain should ship the accounts status at some round levels (how many lisks and votes for accounts) to be really efficient imho.
shuse2
LIP001 - Lisk Improvement Proposal 1 (Improving network synchronisation - decentralised blockchain snapshots)
Topic has been raised in issue #232. Currently syncing new node from beginning is totally unstable, even when snapshots are pretty new it still takes lots of time. Process become very slow, syncing one old block sometimes takes even more time than network produce new block which makes node impossible to catchup with network, sometimes syncing suddenly can speed up for a while to import couple blocks per second, anyway currently syncing is bugged but i know team is working on that i have also saw some new commits in regards to that. Currently lisk team occasionally provide blockchain snapshot stored on lisk.io. Which is great way to remove some unnecessary load from nodes on cpu-level but it's also centralised and relay on one operator.
I think official snapshots should be removed and replaced with decentralised snapshots. Blockchain snapshots is a great idea to improve syncing efficiency. Optionally node operator could allow distribution of blockchain snapshots. Blockchain splitted to packages of - for example 256 or 512 or 5000 blocks, or packs of 128mb with N blocks depending on block size. Each pack can be zipped or compressed with any other maybe more efficient method. It should save some unnecessary cpu load and bandwidth and improve syncing new nodes but it will require more space from node operator so it should be as optional feature and possibly as service in upcoming feature "Standby delegate Market". Each node which is willing to help network sync at the maximum possible level of speed and stability could simply add one flag in config which allow to create and distribute "decentralised snapshots". Each N blocks new package is being created and newly introduced node can simply at efficient speed acquire all packages, uncompress and start rebuilding and verification while another one is downloading from this or another node. But similar solution could be achieved with improvements to network sync. This is proposal to replace official snapshots with decentralised snapshots or possibly improve entire network syncing speed and stability to "snapshot level" and logging while local blockchain height is more than 200 blocks behind network could me improved to more readable format like "Importing blocks 0-256".