Re-architecture/Major refactor for modern Parity

[rphmeier]

Motivation:

Parity was originally written with purely the Ethereum chain in mind -- basic synchronization and mining, with an RPC layer over-top of it. We have now evolved numerous consensus engines, each with their own demands. The light client is on the way and warp sync complicates things further. We've grown a UI which adapts to the operating mode of the client. While developing these features, we have taken on a not-insignificant amount of technical debt. These are the key issues which I feel we currently face:

• Monolithic and non-general BlockChainClient
• Gigantic ethcore crate which is a bottleneck for compilation.
• Strong coupling between client and Miner
• Strong coupling to PoW consensus
• Abstraction over light and full nodes (or perhaps Light, Full, and Archive modes)

Common Transaction Pool Interface

• The light client introduces its own transaction pool (of purely local transactions) which has a high degree of common ground with the full-fledged TX pool.
• Get ready transactions
• Insert local transaction
• Insert remote transaction(s)
• Gas price calibration (could even be done with a Calibrator trait which the TXQ owns)
• Punishing senders
• Pruning invalid transactions.

Decouple Transaction Queue from Miner/Client

• Miner takes an &Mutex<TransactionQueue> as a parameter to updating sealing.
• Mining really has nothing to do with inserting transactions into the queue, only inspecting current queue state and draining transactions to seal into a block.

Decouple Client and Miner

• The two-way referencing between Client and Miner doesn’t represent their roles correctly.
• If anything, Miner should hold a reference to Client, but this isn't strictly necessary.
• update_sealing requires an &mut TransactionQueue, and the ability to fetch latest state, env info, and uncles.
• Submitting seals only requires returning a Result<SealedBlock, _>, not importing to the client directly.
• Chain notification updates can be handled by a higher-level ChainNotify implementation which ties together an Arc<TransactionPool>, Arc<Miner>, and an Arc<BlockProvider> to update the pool with retracted TXs

Decouple Miner and Consensus

• Miner -> Sealer as a name change for clarity. Main job is to prepare blocks for sealing, accept seals for blocks, and request generated seals from the consensus engine.
• update_sealing returns enum: AwaitingSeal, BlockProposal(proposal) which would then be broadcast, rather than triggering broadcast immediately.

Client Traits Refactoring

• Make BlockChainClient: BlockProvider to reduce duplication. Requires BlockId match within BlockChain but that’s fine.
• The core of BlockChainClient is to provide an inspectable State at a given BlockId. This State object should be a publicly-facing encapsulation of ethcore::state::State.
• ChainImport trait for queuing blocks, importing sealed blocks, queue management.
• From a single method we will be able to provide default implementations of nonce, balance, call, estimate_gas, etc. without duplication of code.
• Logs/Tracing doesn’t fully require its own trait, only relevant from RPC layer where we can hold concrete types.
• MiningBlockChainClient can be removed with the above-mentioned Sealer changes.
• The BlockChainClient trait will no longer be sufficient for syncing. But another IPC-enabled trait which wraps a Client, ChainImport, and TransactionQueue would be. This trait could reside entirely within ethsync, allowing IPC codegen to run there exclusively.

Generalize Client

• Two client structs are necessary due to serious differences in DB schema and capability.
• ChainImport can have a BlockLike associated type to reference the verification queue type.
• Unification for Informant necessary (likely simply via slight modification of ClientReport)
• Unification for Service desirable — to move responsibility of DB initialization to the service we would need a trait for describing the schema and associated data. For full nodes this is the SnapshotService. For light nodes it might be the canonical cache (although this as well could reside within the client).

Consensus Engines

• May require broadcast of messages on various stimuli. Should be done via a “Broadcast Endpoint” or similar trait rather than EngineClient: this has nothing to do with a block chain client.
• Similarly, the sync provider shouldn’t route consensus messages through the Client.
• May require signing with a specific account on various stimuli. EngineSigner is a good step forwards, but it should be set not via Sealer but through the engine directly. Decouple Sealer from AccountProvider.
• May require queries of current state. Can be done via a BlockChainClient, but should not be necessary for purely verification-based operation (considering light nodes).
• With the right traits, we can have the Engine abstraction reside below Client in the crate hierarchy, with implementations in their own crates.

What does this enable?

• Clear separation of concerns.
• Better abstractions for easier-to-understand and more testable code.
• Easily split up components into their own crates.

cc @ethcore/core-devs for feedback.

[rphmeier]

(this is sort of a stream-of-consciousness write-up, just going through the things which seem the most fragile/hacky in the core code for now)

[debris]

I couldn’t agree more! I would also add decoupling BlockChainClient and database. Our tests, where it's not necessary, should not use io at all

[rphmeier]

@debris Good point, that's done in #4566.

Also relevant: #4591

[5chdn]

What's the take on this?

[debris]

I will close this pr as most post of the points have been addressed by independent pull requests.