We currently have large (> 1Mb) gossip messages for foreign proposals. Gossipsub is not designed for large messages.
In the case where a large amount of data must be transferred between many nodes, we've used the gossip and request strategy, which entails gossiping a succinct notification followed by an explicit request for the data.
Foreign proposals contains a block, a QC and substates. Additional data may be required to ensure authenticity (e.g. a merged merkle proof of the applicable substates) but this is out of scope for this issue.
Proposal 1
Notify applicable foreign shard groups about a foreign proposal on each shard group's topic. The notification includes the block id and the shard-applicable transaction IDs.
Each receiving node selects a member of the foreign committee (this may be random, or may be based on the relative index of the two nodes to reduce traffic to a single node)
The node sends a ForeignProposalRequest(req_id) to the selected node
The sending node streams responses containing the applicable data. Since substate sizes may exceed framing limits, these responses are broken up.
The receiving node assembles the messages using req_id and when the stream is complete, passes the payloads to the foreign proposal handler.
Protocol details:
req_id is a numeric identifier that is unique to the sender and receiver peer. Duplicate req_ids between different node pairs must not interfere with other node pairs. i.e. it is purely informational and should be tracked locally per request.
The request contains the req_id, and shard-applicable transaction ids
The response must contain the req_id, the response type and the payload
Edge cases:
For simplicity, a stream which fails midway can be discarded and a new request sent.
tentative: If a local proposal is received that contains a reference to an unknown foreign proposal, the local node may request the foreign proposal from the local sending node.
If a request times out, another foreign node must be selected
we may wish to include various response types to allow for edge cases. The initial implementation should prioritise simplicity so that we can evaluate which further improvements are necessary.
Additional benefits:
If a transaction is in LocalPrepared or LocalACcepted for some timeout, we can use the same mechanism to request an expected missing foreign proposal from the foreign node. If a foreign node is unaware of any of the requested transactions, it may request and sequence the previously unknown transactions.
Problem
We currently have large (> 1Mb) gossip messages for foreign proposals. Gossipsub is not designed for large messages. In the case where a large amount of data must be transferred between many nodes, we've used the gossip and request strategy, which entails gossiping a succinct notification followed by an explicit request for the data.
Foreign proposals contains a block, a QC and substates. Additional data may be required to ensure authenticity (e.g. a merged merkle proof of the applicable substates) but this is out of scope for this issue.
Proposal 1
ForeignProposalRequest(req_id)to the selected nodeProtocol details:
Edge cases:
Additional benefits: If a transaction is in LocalPrepared or LocalACcepted for some timeout, we can use the same mechanism to request an expected missing foreign proposal from the foreign node. If a foreign node is unaware of any of the requested transactions, it may request and sequence the previously unknown transactions.