[Epic] Improve the synchronization process

jakubcech commented 5 years ago

Description

We should see how we can rework the synchronization process experience in IRI. We can rethink the current from the tips approach.

We can explore the hypersync idea that Luca already has shelved somewhere probably, or devise something different.

Motivation

The current synchronization process takes too long and the node operator experience (NOX) isn't great. There's no good concept of progress either, you can be stuck on one milestone X thousands of milestones back for days and then suddenly the node fully syncs.

The process probably also still gets stalled completely in certain situations because it sometimes tends to pick up after a restart.

Requirements

[x] I'm able to see the synchronization progress so that I know the node has not stalled.
[x] I'm able to see an estimate of how long the estimation process will take. Smth like exponential moving average will be enough imo.
[x] I'm able to see the progress of synchronization in %. Can be done just on startIndex, latest and current subtangle (?).
[x] I'm able to see that the node has stopped syncing for a certain amount of time. The node reports a message hinting that.
[ ] I'm able to synchronize 1 day of TXs at 10 TPS in 15 minutes.

jakubcech commented 5 years ago

This might all be 'addressed' now with this so we should see: https://github.com/iotaledger/iri/pull/1530

GalRogozinski commented 4 years ago

Also #1575 added a progress bar

iotaledger / iri