feat(mempool)!: gossip reference to tx and request if needed

[sdbondi]

Description

• Gossip full transaction if less than 64Kb
• Otherwise, gossip new transaction notification
• Batch notifications and request transactions using mempool sync in batches every interval
• Raised the message size limit for base node GRPC

Motivation and Context

Transactions routinely are too large for gossipsub. A new protocol has been added to mempool sync. Many transactions may be gossiped in quick succession and therefore it is inefficient to request each time a transaction notification is received. This protocol aims to address this by collecting notifications and periodically requesting many transactions in batched.

Details:

1. NewTransaction gossip message is received,
2. If it contains the full transaction (it fits into the max message size for gossipsub) the transaction is added as before.
3. Otherwise, the references to new transactions, the propagation peer and message id are collected
4. Every interval the batch is processed, requesting all transactions in a single request
5. Transactions are streamed back and validated.
6. The result of each of these is collected and the propagation message is either accepted, rejected or ignored depending on the result. This ensures that further propagation of the transaction message only occurs for valid transactions.

Base node GRPC decode and encode limit were raised from 4MiB to 10MiB, this is because some blocks within the legal weight limit were not transferable via GRPC when mining (~6MiB) causing GRPC errors and the inability to mine.

How Has This Been Tested?

Manually, submitting transactions and observing logs

What process can a PR reviewer use to test or verify this change?

Submit transactions to one node and observe the mempool of other nodes on the network

Breaking Changes

• [ ] None
• [ ] Requires data directory on base node to be deleted
• [ ] Requires hard fork
• [x] Other - Please specify BREAKING CHANGE: Changes to the mempool sync protocol are not compatible with previous versions

[github-actions[bot]]

Test Results (CI)

    3 files    126 suites   12m 6s ⏱️ 1 160 tests 1 160 ✅ 0 💤 0 ❌ 3 480 runs  3 480 ✅ 0 💤 0 ❌

Results for commit ad35c63d.

:recycle: This comment has been updated with latest results.

[github-actions[bot]]

Test Results (Integration tests)

 2 files   1 errors  9 suites   1h 34m 47s ⏱️ 18 tests 13 ✅ 0 💤  5 ❌ 27 runs  14 ✅ 0 💤 13 ❌

For more details on these parsing errors and failures, see this check.

Results for commit c78fac23.

:recycle: This comment has been updated with latest results.