Transaction sizes can grow too large

Context

This issue is a continuation of https://github.com/informalsystems/hermes/issues/2993. That particular issue was closed since the packet was successfully cleared. We'd like to avoid this situations in the future, however. This will involve a cross-repository and cross-organizational effort, which I'll track here.

Next steps

After a discussion with the IBC-go team and SDK we remarked on the following:

The proposer likely bypassed the Tendermint-side check on the max size of this tx.
Most likely, the solution would not live in IBC-go.
Marko: we want multiple levels of protection:
- Tendermint shoulnd’t allow >1 MB tx in mempool.
- SDK either shoulnd’t allow >max_bytes part of their gRPC API. By default, 4MB. Should be validated in CheckTx.
- The limiter in the field should be module-specific (IBC-go).

Action items:

Adi: Double-check that indeed the Tendermint-level protection on the TX size can be bypassed
Jacob G: Check with CosmoStation validator (who proposed the offending tx) to investigate if their Tendermint configuration was by design loose to allow such a large transaction to enter their mempool and the blockchain.

Acceptance criteria

To close this issue, the following are necessary

[x] identify culprit and causes for why this problem appeared
- see the two action items above & comments belove: it appears that Tendermint-level protection on the TX size can be bypassed
[ ] identify next steps -- for example, engineering efforts or additional investigation, measures for future protection -- if any are needed by any team (SDK, Notional, IBC, Tendermint, Hermes)
[ ] for each each team that has next steps identified, there is an open issue to track that work in the team's repo

Follow-up on: Adi: Double-check that indeed the Tendermint-level protection on the TX size can be bypassed

I did a quick check using tm-load-test and was able to submit to the mempool a ~9.6MB transaction using:

Terminal 1:

./build/tendermint node --proxy_app=kvstore

Terminal 2:

% ./build/tm-load-test -T 3 -r 1 -s 10085760 --endpoints ws://localhost:26657/websocket

I assessed that the transaction entered successfully in the blockchain based on these logs from Terminal 1

I[2023-01-11|15:05:25.455] WSJSONRPC                                    module=rpc-server protocol=websocket remote=127.0.0.1:58109 method=broadcast_tx_async
I[2023-01-11|15:05:25.497] Timed out                                    module=consensus dur=924.466ms height=1360 round=0 step=RoundStepNewHeight
I[2023-01-11|15:05:25.619] received proposal                            module=consensus proposal="Proposal{1360/0 (217EDC12B24A2459D424003DEDA195C2B90D542DB7A96E7E91C06719D096ADAE:154:86B82EB05665, -1) 249C22F218E5 @ 2023-01-11T14:05:25.601088Z}"
I[2023-01-11|15:05:26.450] WSJSONRPC                                    module=rpc-server protocol=websocket remote=127.0.0.1:58109 method=broadcast_tx_async
I[2023-01-11|15:05:26.940] Client closed the connection                 module=rpc-server protocol=websocket remote=127.0.0.1:58109
I[2023-01-11|15:05:26.940] service stop                                 module=rpc-server protocol=websocket remote=127.0.0.1:58109 msg="Stopping wsConnection service" impl=wsConnection
E[2023-01-11|15:05:26.940] error while stopping connection              module=rpc-server protocol=websocket error="already stopped"
I[2023-01-11|15:05:28.487] received complete proposal block             module=consensus height=1360 hash=217EDC12B24A2459D424003DEDA195C2B90D542DB7A96E7E91C06719D096ADAE
I[2023-01-11|15:05:28.498] Timed out                                    module=consensus dur=3s height=1360 round=0 step=RoundStepPropose
I[2023-01-11|15:05:28.547] finalizing commit of block                   module=consensus height=1360 hash=217EDC12B24A2459D424003DEDA195C2B90D542DB7A96E7E91C06719D096ADAE root=9803000000000000 num_txs=1
I[2023-01-11|15:05:28.710] executed block                               module=state height=1360 num_valid_txs=1 num_invalid_txs=0
I[2023-01-11|15:05:28.728] committed state                              module=state height=1360 num_txs=1 app_hash=9A03000000000000
I[2023-01-11|15:05:28.876] indexed block exents                         module=txindex height=1360
I[2023-01-11|15:05:29.547] Timed out                                    module=consensus dur=688.591ms height=1361 round=0 step=RoundStepNewHeight
I[2023-01-11|15:05:29.676] received proposal                            module=consensus proposal="Proposal{1361/0 (EE824124C5D596D424F056D7129EBEAF5FBC11E40E37FA286C23D1B5B91061FB:154:BD8513E08E77, -1) E0CEEB1E090E @ 2023-01-11T14:05:29.656957Z}"
I[2023-01-11|15:05:32.547] Timed out                                    module=consensus dur=3s height=1361 round=0 step=RoundStepPropose
I[2023-01-11|15:05:32.597] received complete proposal block             module=consensus height=1361 hash=EE824124C5D596D424F056D7129EBEAF5FBC11E40E37FA286C23D1B5B91061FB
I[2023-01-11|15:05:32.654] finalizing commit of block                   module=consensus height=1361 hash=EE824124C5D596D424F056D7129EBEAF5FBC11E40E37FA286C23D1B5B91061FB root=9A03000000000000 num_txs=1
I[2023-01-11|15:05:32.805] executed block                               module=state height=1361 num_valid_txs=1 num_invalid_txs=0
I[2023-01-11|15:05:32.825] committed state                              module=state height=1361 num_txs=1 app_hash=9C03000000000000
I[2023-01-11|15:05:33.001] indexed block exents                         module=txindex height=1361

The above would not work successfully on a vanilla setup. To make it work, I had to change the following Tendermint configuration params:

# Maximum size of request body, in bytes
max_body_bytes = 100000000
# %% ADI CHANGED ^ from 1000000

# Maximum size of request header, in bytes
max_header_bytes = 10485760
# %% ADI CHANGED ^ from 1048576

# Maximum size of a single transaction.
# NOTE: the max size of a tx transmitted over the network is {max_tx_bytes}.
max_tx_bytes = 10485760
# %%% ADI CHANGED ^ from 1048576

informalsystems / interchain