Separate processes for p2p blockchain sync and state creation

[vogel76]

Logical split of steemd process to separate process doing pure blochain sync (p2p) and another one, doing state sync basing on previously synced local blockchain storage. This separation will eliminate state processing influence on blockchain sync.

[mvandeberg]

Please explain you're thinking of what this change will look like.

I am confused by "This separation will eliminate state processing influence on blockchain sync." The p2p code already runs in a separate thread. And if you have the p2p code too far out of sync with state processing then you run the risk of accepting bad blocks. The current p2p code requests blocks in bursts. By default, it is configured the prefetch 10k blocks. Syncing logs every 10k blocks and is significantly slower than a reindex due to the creation of undo states on sync.

I 100% believe p2p code is bottlenecked by blockchain state, but the state processing has to be there and so long as the p2p code is not bottlenecking state processing, there is not an immediate need to move p2p out of process.

I do want to do this eventually to make the architecture more modular, but I do not believe performance is a short term motivator for this change.

[vogel76]

@mvandeberg This task is an effect of ideas you proposed on slack some time ago. I generally like this idea because it clarifies duties of each module. Anyway I was surprised that p2p (at least p2p_plugin which is some extension over pure p2p) has so strict dependency on state. I thought p2p shall just do data transfer without any deeper analysis. Our intention was to move data input coming from p2p and allow it to be handled directly by this separate tools. So data flow could look as follows: 1) p2p-sync tool retrieves input blocks and stores them in local file (and does same data transfer to other p2p clients as it is currently) 2) steemd periodically analyzes file state (it can be also explicitly signaled). It processes all new blocks accepted ones are put into actual blockchain file and sent back to p2p network by directly integrated p2p module in same process. This way we avoid potential blocking p2p by slow state processing (now it could be also possible by malicious API calls locking database what finally can lead to blocking p2p thread). I have heard similar solution (separating p2p sync and actual processing) is done in bitcoin network (don't know details unfortunetely). The question is if really p2p communication can be broken by slow state processing. Maybe simplest is to make explicitly slowed down version of steemd (where state processing is slowed ie by sleep or similar construct) and check it communication problems appear...

[mvandeberg]

It is not entirely true that the p2p code is not dependent on state. All the p2p code needs is the validity of the block/transaction it is relaying. This result, however, is state dependent. So while you could have a p2p protocol that grabs every single known block before pushing it to the blockchain, this is dangerous. A malicious seed node could send 1,000,000,000 fake blocks that the p2p code cannot verify. If it ran ahead unbounded then a remote seed could cause all sorts of problems.

We know that there are some p2p problems when a node is bogged down by too many API requests. We have yet to see any evidence of this behavior of a node in normal operation. Appbase addresses this problem directly. We gain no reindex improvements from optimizations to the p2p code. While doing some refactoring to the p2p code is something that we would like to do, it is not a priority at this time.

Moving this to the backlog for now.