Validate consensus implementation in the presence of network issues

[dinowernli]

Rough idea:

• Implement a custom channel for RPCs (see [1])
• Add random fault injection to RPCs within what the protocol fault-tolerance should be able to handle
• Use the existing diagnostics to validate that all nodes agree on leader, contents, versions, etc

--

[1] https://github.com/hyperium/tonic/blob/master/examples/src/mock/mock.rs#L35

[dinowernli]

More info: https://users.rust-lang.org/t/how-to-wrap-a-tonic-streaming-operation-in-a-trait/75257/13