Reduce delay in roothash commit processing

[kostko]

Since Tendermint blindly schedules when blocks are proposed, the following can happen, causing unnecessary delay while processing batches:

2019-05-21T10:08:29.299999503Z commit called on node 1
2019-05-21T10:08:29.299999503Z tx roothash commit from node 1 (->h=25)
2019-05-21T10:08:29.30041168Z  enter propose (h=25)
2019-05-21T10:08:29.382282441Z tx roothash commit from node 2 (->h=26)
2019-05-21T10:08:29.385026717Z tx roothash commit from node 3 (->h=26)
2019-05-21T10:08:29.389549228Z tx roothash commit from node 4 (->h=26)
2019-05-21T10:08:29.719338595Z executed block (h=25)
2019-05-21T10:08:29.724729696Z enter propose (h=26)
2019-05-21T10:08:30.20912014Z  executed block (h=26)

Basically the first commitment causes a new Tendermint block to be proposed and executed, containing only the one transaction corresponding the the first roothash commit. The subsequent three commits that arrive just 80ms later are put into the next block and the whole roothash round is delayed by ~500ms.

Possible solutions:

1. Broadcast commits between compute committee members via p2p. Then each committee member can push them in bulk to Tendermint once it sees enough of them or a timeout elapses (duplicate transactions will be ignored by mempool anyway).
2. Implement a smarter Tendermint mempool (possible in latest Tendermint develop branch as Mempool is now an interface) which allows ABCI apps to emit some scheduling information from CheckTx based on which it could wait "a bit" before releasing any transactions. This seems much more complex and may be less effective (as the mempool can also contain a bunch of non-roothash transactions).
3. Something else.

If we decide to do this, I would be in favor of (1) as it keeps this functionality under our control and avoids the complexity of implementing (2).

In any case, these effects probably cannot be completely eliminated so as a mitigation we should make sure that:

• We can process large batches of transactions to reduce the effect of Tendermint latency on throughput.
• We implement something like batch pipelineing (#1581).

[kostko]

Now that #1749 introduced a commitment.Pool which abstracts most of the discrepancy detection logic, I can imagine a smart mempool that uses it to determine when to trigger new consensus rounds. Of course it also needs to consider other factors like the presence of any non-roothash transactions and needs to make sure not to stall consensus.