mempool: explore how tendermint mempool behaves with large Tx

[liamsi]

One thing we haven't really thought about yet: LazyLedger (app) messages could potentially become quite large. I'm not sure if the tendermint mempool in its current form will able to cope with these.

We should investigate:

• [ ] how well can the tendermint mempool deal with large Tx?
• [ ] could Tx gossiping pose any security issue?
• [ ] how should we cap the Tx size if it could cause a problem?
• [ ] how large do we expect ORU blocks to be that will be posted onto LL? Quoting John:

  Just as a ballpark, 1 MB messages should probably a good starting point upper bound I think

Maybe @ValarDragon @marbar3778 have any insights on this.

related: tendermint/tendermint#7918

[ValarDragon]

IIRC, the Tendermint p2p stack places max message sizes on every message, not sure what the defaults are set to for the mempool.

There is also a max bytes / tx check in the mempool, used here: https://github.com/tendermint/tendermint/blob/master/mempool/mempool.go#L109. This is inferred from the blocks size. https://github.com/tendermint/tendermint/blob/302aec6dcce639763541717063398615f5b1b32e/state/tx_filter.go#L12

Beyond that, the mempool cache maintains a hashmap from H(tx) to (is tx present). So all recipients of a tx will have to compute a large hash. Then the message will be passed to the state machine via ABCI for a CheckTx.

At the moment, Tendermint places no punishment on peers who send you invalid messages (relevant line https://github.com/tendermint/tendermint/blob/master/mempool/reactor.go#L182), so this exposes a spam vector where I as a peer can make you perform tons of hash operations & lots of CheckTx's. These actions are only on a RLock mutex that is shared with consensus, so it could potentially delay consensus as well. None of this is really unique to the large message setting, just the large message setting potentially increases the delay per tx / delay for things under the mutex.