(new WebTransport(protocol)).pipe

[Ayms]

Maybe I am misreading the specs or misunderstanding the intent but new WebTransport(url,...) seems not generic enough

I would better see new WebTransport(protocol)

And then (new WebTransport(protocol1)).pipe(new WebTransport(protocol2)...pipe(new Webtransport(quic)

Maybe take a look at https://github.com/Ayms/node-Tor#evented-pipes-phase4 and https://github.com/Ayms/node-Tor/tree/master/docs which allows simply (among others):

protocol.on(‘data’,//process_data)
protocol.send=//send data

Piping with node-Tor is as simple as:

piping(protocol) //turn protocol into a Duplex object
protocol._write_b=//process_data from node-Tor
protocol.send=function(data) {this._stream_f.push(data)} //send data to the Tor protocol
protocol.pipe(node-Tor)

And <any protocol>.(...).pipe(<any protocol>).pipe(node-Tor).pipe(RDV peer)

Where piping is similar to WebTransport and then we can have:

(new WebTransport(bitcoin)).pipe(new WebTransport(node-Tor)).pipe(new WebTransport(quic/webrtc))

or

(new WebTransport(protocol)).pipe(new WebTransport(ws)).pipe(new WebTransport(quic))

or whatever you like

[jan-ivar]

Maybe I am misreading the specs or misunderstanding the intent but new WebTransport(url,...)

Hi @Ayms, the goal is for JS in a browser client to establish an HTTP/3 UDP connection to a server at url. The number of protocols was reduced to 1 last week.

I confess I'm a bit confused. What would pipe do on a client (and e.g. what is bitcoin in this example)?

(new WebTransport(bitcoin)).pipe(new WebTransport(node-Tor)).pipe(new WebTransport(quic/webrtc))

[Ayms]

So WebTransport is "just" HTTP3, how does this work to do WS-like APIs as described in the target of the spec?

The idea here could be that url could be a peer too (like WebRTC) and that you could pipe whatever protocol ending to quic/HTTP3 (quic layers) or WebRTC

In my example bitcoin is the bitcoin protocol and node-Tor the Tor protocol, implemented inside browsers (like https://github.com/Ayms/bitcoin-transactions), please see the links above

Or whatever protocols and combination of them in fact, independently of what meaning we give to the term "transport"

[jan-ivar]

how does this work to do WS-like APIs as described in the target of the spec?

Are you referring to the asbtract: "implementing pluggable protocols underneath with common APIs on top"?

If so, this is a document we inherited from WICG, which we're working on to satisfy our charter, which I think more properly captures our target (see mission and scope).

The original design of multiple protocols in one API was challenged last week when the IETF WebTransport WG decided to standardize only one protocol: HTTP3.

Once the call for consensus on that decision has completed, we'll meet to update our document to reflect that decision, including hopefully clarifying the abstract.

The idea here could be that url could be a peer

Peer-to-peer is out of scope in our charter. But that shouldn't preclude other groups from tackling it.

and that you could pipe whatever protocol ending to quic/HTTP3 (quic layers) or WebRTC

We could probably do a better job separating scope from API design. A "common API" !== common constructor. https://github.com/w3c/webtransport/issues/180

Whichever group decides to tackle another native browser protocol, would probably benefit from having its own constructor with sensible arguments for that protocol, as would users. Differently constructed APIs can still share "common" patterns and behaviors, maybe even allowing for common concepts like "pipes" in the future, if that makes sense.

This spec defines various mixins other specs may reuse if they wish to build on the browser functionality WebTransport exposes, as well as embracing standard streams (which come with their own concept of piping), so in that sense this API hopefully contributes to establishing common API patterns in this space.

[Ayms]

I am referring to "Lack of WebSocket-like API without head-of-line blocking" in the webtransport root git repo

I think to summarize what I mean that WebTransport should allow to easily pipe bidirectionnal streams to the quic layer, via standard streams as you mention and/or some kind of duplex object to ease this (like mine in node-Tor), then maybe a pipe method for webtransport itself is not required

Then HTTP(3) would just be one of the protocols that can be piped, right now I don't understand very well the target of such API with only one protocol and it seems like it is not designed to allow what I am suggesting

[jan-ivar]

I am referring to "Lack of WebSocket-like API without head-of-line blocking" in the webtransport root git repo

Functionally, WebSockets is TCP to/from a server, whereas WebTransport is UDP to/from a server, solving head-of-line blocking. WebSockets don't permit other protocols AFAIK, unless by "protocol" you mean custom app framing in your payload?

On the API shape, WebSockets could be polyfilled on top of WebTransport, but I'd advise against it, since that's like using callbacks instead of promises.

WebTransport should allow to easily pipe bidirectionnal streams to the quic layer, via standard streams as you mention and/or some kind of duplex object to ease this (like mine in node-Tor)

I think I see what you mean, but for the sake of other readers, I see two ways to read "pipe bidirectional streams" here:

1. ✅ Piping to/from bidirectional byte streams https://jsfiddle.net/jib1/cyx15q7h/ which we already do:

   
   const {writable, readable} = new TransformStream();
   const p = readable
   .pipeThrough(new TextEncoderStream("utf-8"))
   .pipeThrough(await transport.createBidirectionalStream())
   .pipeThrough(new TextDecoderStream("utf-8"))
   .pipeTo(new WritableConsole());

const writer = writable.getWriter(); await writer.write(send.value); await writer.close(); await p;

2. ✅ Piping chunks that are streams themselves (aka stream-of-streams). I've polyfilled _stream-of-uni-streams_ in https://github.com/w3c/webtransport/issues/40#issuecomment-767734866. https://jsfiddle.net/jib1/obu4qtce/
```js
const streams = new TransformStream();
const p = streams.readable
  .pipeThrough(transport.unidirectionalStreams)
  .pipeTo(new WritableMessageStreamsConsole("From server: "));

const streamsWriter = streams.writable.getWriter();

// Stream a message
const message = new TextEncoderStream();
await streamsWriter.write(message.readable);
const writer = message.writable.getWriter();
await writer.write(text.value);
await writer.close();
await p;

I think you mean the latter, except for bidirectional streams, which I think is a good idea. I've noted this in https://github.com/w3c/webtransport/issues/40#issuecomment-767742735.

The symmetry alone might suffice, but if you have a use case for this, that might help.

[Ayms]

1 looks to be very exactly what I mean where you can replace new TextEncoderStream and new TextDecoderStream by new WhateverProtocol

For WS I think you should add the support for it, even if not very square, this is what we have today, we can bet that people will want to use it without going to the streams level and turning their "protocol" into a duplex stream

Not sure what you mean by "WebSockets don't permit other protocols AFAIK" but people are using other protocols on top of WS (like node-Tor for the Tor protocol from the browser)

[jan-ivar]

1 looks to be very exactly what I mean where you can replace new TextEncoderStream and new TextDecoderStream by new WhateverProtocol

Great! Though note you'll need 2 to map WebSocket semantics which allow sending a short text message, followed by a 1GB blob, followed by a third text message, whether you handle them arriving out of order or not. See this note.

Not sure what you mean by "WebSockets don't permit other protocols AFAIK" but people are using other protocols on top of WS

"On top of" is fine, and different from "instead of" the protocol used to establish a network connection with a server, which remains HTTP/3 (which is QUIC-based UDP, not TCP btw). Adding another one of those in browsers would come with significant security considerations, and would need to come from the IETF.

For WS I think you should add the support for it, even if not very square, this is what we have today, we can bet that people will want to use it without going to the streams level and turning their "protocol" into a duplex stream

There's definitely a learning curve with streams, but the benefits, like backpressure, composition, sending massive data without massive buffers, and (eventually) performance, I think are more than worth it. IMHO it's the right abstraction for the problem of streaming data in JS. WebSocket just isn't.

I think you're right though that many folks are going to try to avoid learning streams, and I worry those folks are going to have a hard time. As a corollary, I still see folks building "callback pyramids-of-doom" using Promise constructors, because they fundamentally don't understand chaining.

I might get behind an example showing how one might polyfill WebSocket or WebSocketStream in the spec, as an educational tool to help people migrate concepts. This might prove a good place to preemptively explain the shortcomings of doing so.

[Ayms]

If you look back in the past you might see that I am the one that did revive the W3C Streams API, backpressure & co, not even part of the acks...

Anyway, of course quic is udp, streams are easy for gafa like people, not everybody, right or wrong you can't force people to understand what they don't, then they would like to try it ws like, just for backward compability, then a bridge would be needed with webtransport, and flow control is up to them, this is just my opinion... you can close this issue if it's definitely decided that it will not be the case, that would be a mistake I think

[Ayms]

Just to be clear, I am not asking for myself of course, I would use the streams way, just my opinion again...

[jan-ivar]

@Ayms Can you describe what this "WS-like API without head-of-line blocking" would look like?

A WebTransport server wouldn't recognize message framing the way a WebSocket server would, so how would you map e.g. sending a large file? See also https://github.com/w3c/webtransport/issues/49#issuecomment-551067096.

[Ayms]

Can you describe what this "WS-like API without head-of-line blocking" would look like?

This term is from you own description, not mine

A WebTransport server wouldn't recognize message framing the way a WebSocket server would, so how would you map e.g. sending a large file?

What is the issue with piping a ws interface to the quic layer and piping other protocols on top of ws as people do (including transferring large files or whatever), or piping directly the protocols to the quic layer via Webtransport again ? Knowing that the upper layers at client and server side must handle flow control and sync of messages, if I read correctly this is what #49 suggests too

What I am saying is that people will want to use the ws like stuff (even if useless), because they are used to it and it's more simple/backward compatible than redevelopping the entire thing and interface it with WebTransport

I am pretty sure that this will resurface in the future but I might be wrong

[jan-ivar]

if I read correctly this is what #49 suggests too

https://github.com/w3c/webtransport/issues/49#issuecomment-521817167 says "I do not suggest that a QUIC transport should implement any of these interfaces."

What is the issue with piping a ws interface to the quic layer ... Knowing that the upper layers at client and server side must handle flow control and sync of messages

The upper layers can do that today.

But since Quic itself does not support built-in message framing, it seems cleaner to leave this to specific servers and apps instead of having this spec dictate a canonical framing format that all WebTransport servers would have to implement (and somehow negotiate?), solely to support a legacy API that doesn't have back pressure.

[jan-ivar]

Then HTTP(3) would just be one of the protocols that can be piped,

To clarify: the data being piped is still raw bytes. We're not limiting what can be built on top.

[jan-ivar]

While we're not ruling out this being added in the future, this seems like something applications can polyfill now. If there's a lot of demand, we could reopen this. Otherwise I think we can close this like we did #49.

To move this forward, the IETF would need to add message framing, so that might be the right place to advocate for this.