Multiplexing: Stream prioritization

[zmanian]

One of the requirements to use libp2p for Tendermint's peer to peer level is to support prioritized mulitplexed channels.

Tendermint's p2p layer currently provides this functionality. This is used to ensure vote carrying messages, block part carrying messages and peer consensus state queries and responses have priority over other messages like transaction gossip.

The desired api is that a subroutine of the consensus system like Tendermint reactors should be able to request a channel from libp2p with desired priority level.

A more advanced API would allow a subroutine to request channels with even more application specific semantics like datagram semantics, datagram + error correcting code semantics or set reconciliation semantics..

Chains like Solana and Harmony are experimenting with these kind of channels in the p2p layer.

[ebuchman]

Here are some references to implementation in case it helps:

• selecting a channel to send on according to priority: https://github.com/tendermint/tendermint/blob/v0.32.2/p2p/conn/connection.go#L498
• initializing priorities: https://github.com/tendermint/tendermint/blob/v0.32.2/consensus/reactor.go#L135

Note the prioritization only applies to sends but not to receives.

[raulk]

@marten-seemann Does QUIC support prioritised streams natively? Or is this feature only specified in HTTP/3 semantics?

Re: channel-specific concerns, I agree. Another one we've discussed is stream compression.

[marten-seemann]

Fun fact: The QUIC working group just removed priorities (more context) from the HTTP/3 protocol today. The intention is to come up with a simpler prioritization scheme than the immensely complex priority tree that HTTP/2 defined soon, hopefully before QUIC ships as a RFC.

Every QUIC implementation (really, this applies to every stream multiplexer) is free to implement a prioritization logic and expose an API to set those priorities. This is not a trivial task, as it also involves setting the stream scheduling logic: Currently quic-go uses round-robin for scheduling, but for this might not be the best scheduling logic if you're dealing with high-priority streams.

The next question is if you need to communicate priorities on the wire. This is what the HTTP/2 priority scheme was (trying to) achieve. From a quick glance at the code linked in this issue, it seems to me like the priorities here are local properties, and don't need to be communicated on the wire. Is that correct?

[ebuchman]

it seems to me like the priorities here are local properties, and don't need to be communicated on the wire. Is that correct?

That's correct

[djrtwo]

Certainly could be valuable to eth in the long term. In initial phases of eth2, all messages are consensus messages so we get a pass for a bit, but once user level activity is added this would be a simple solution that might be sufficient for keeping the consensus message propagation quick.

Alternatives we might look into is a privileged subnet of validators to be able to very efficiently pass consensus messages. There are some interesting privacy preserving zero knowledge schemes, but out of scope for this issue.