Elaborate on claim that HTTP/2 does not multiplex full-duplex streams

[glyn]

The Warp concept page states:

although HTTP/2 introduces a limited form of multiplexing, it multiplexes RPC calls, not full-duplex streams

but I'm not sure this is true. For instance, protocols such as ~rsocket~ gRPC provide bidirectional streams using HTTP/2. It would be helpful if the Swim docs could elaborate on the basis for the above statement.

[adamkewley]

@glyn, looking at the HTTP/2 spec it seems like, yes, you can stream via a HTTP/2 stream by having the server send a stream of DATA frames back to the client (effectively, this was called chunked-encoding in HTTP/1.1) but, as specified, it seems like the order of events must be:

• Client initates a new HTTP/2 stream (note: server can't)
• Client sends a full request (can be streamed)
• Server sends full response (can be streamed)

It's not clear, from a brief reading, that the request can occur at the same time as the response. So, it's streaming in both directions (good) but not necessarily bidirectional, unless I misunderstood the spec?

(note: I'm not from swimai, they're just on my watch list)

[glyn]

According to 5.1.1 Stream Identifiers both client and server can initiate new streams. According to 5. Streams and Multiplexing each stream is a bidirectional sequence of frames.

[adamkewley]

@glyn wow, I learn something new every day :+1:, is the reason this isn't done often (yet) because HTTP clients in the wild still assume a request + response cycle?

[glyn]

Some HTTP clients such as gRPC use the bidirectional streaming facilities of HTTP/2. I don't know whether that counts as "often". ;-)

[glyn]

@c9r would you or another Swim contributor care to answer from Swim's perspective?

[c9r]

The core issue has to do with continuing to send request frames whiles concurrently receiving response frames (i.e. full duplexing). HTTP/2 is full duplex across streams, in that one stream may be sending while another is receiving. But HTTP/2 is still very much half duplex within a given stream. gRPC breaks with the spec in this regard, which is fine.

It seems the compliant way to do full duplex streaming over HTTP/2 is still to use WebSockets (RFC 8441. WebSockets over HTTP/2 will give you legit full duplex multiplexing over HTTP/2 streams, and should have broad browser support soon enough. Though with the advent of QUIC and HTTP/3, WebSockets starts to look jenky and unnecessary when encapsulated in HTTP/2 data frames.

The deeper issue with full duplex streaming is what's at the endpoints of the stream. A stateful application layer becomes essential. And you need to be able to demux streams and route them to granular endpoints within server clusters. This is where Swim comes in; it solves the problem of scaling a stateful application architecture that can efficiently maintain distributed real-time coherency.

We'll jump on the QUIC HTTP/3 bandwagon as soon as its viable. At this stage, we haven't been able to improve upon doing our own multiplexing within WebSockets. We've tried quite a few alternatives, including some over raw HTTP. We can make it work Swim to Swim just fine. But it's always fallen short when interacting with other implementations, in particular Web browsers.

[c9r]

WARP is primarily a semantic model for streaming cache coherency between stateful application objects (Web Agents). WARP is an alternative model to RPC, and by extension, REST. RPC is a fundamentally inconsistent primitive on which to build distributed systems.

The WARP coherency model can be implemented over many transports, just as the RPC model can. The current WARP wire protocol serves to backport streaming cache coherency into existing systems, chiefly Web browsers. It’s the coherency model we really care about and lean on; we’ll tunnel through whatever transport layers we need to in order to bootstrap a stream coherent Web.

Better documentation of the WARP coherency model is obviously essential to that endeavor. We’re hoping to release a detailed spec and paper by the fall.

[c9r]

Let’s use this as a tracking issue for improving WARP documentation.

[glyn]

Thanks for the explanation. I like the idea of further docs.

[davedoesdev]

Where does the spec say half duplex within a given stream?

https://tools.ietf.org/html/rfc7540 section 5:

A "stream" is an independent, bidirectional sequence of frames exchanged between the client and server within an HTTP/2 connection.

• A single HTTP/2 connection can contain multiple concurrently open treams, with either endpoint interleaving frames from multiple streams.
• Streams can be established and used unilaterally or shared by either the client or server.

[nitely]

Http/2 does support full-duplex communication over a single stream.

A server can send a complete response prior to the client sending an entire request if the response does not depend on any portion of the request that has not been sent and received.

https://datatracker.ietf.org/doc/html/rfc7540#section-8.1

It's indeed what gRPC does. Even WebSockets run over http/2 now.