Low latency streaming with WebTransport + WebRTC

[jianjunz]

The use case is low latency realtime live streaming from server to browser. The server could be a SFU or a cloud gaming server.

Nowadays, we can achieve it by using WebRTC 1.0 or WebTransport + MSE, but both of them are not perfect. WebRTC involves ICE which is not required by server-client architecture. WebTransport + MSE, more specifically, reliable QuicStream + MSE, may have large latency than WebRTC when network is not good.

Here are two ideas to reduce the latency

• A PeerConnection accepts a plugable transport module, so it could be an RTCDtlsTransport defined in webrtc-pc or a DatagramTransport defined in WebTransport. In this case, a new transport interface is needed.
• Expose RTP module to JavaScript developers to fulfill the gap between WebTransport and WebCodecs. Then, applications could use DatagramTransport + RTP module + WebCodecs to receive and decode media streams.

[aboba]

Currently WebRTC offers support for advanced video features (e.g. simulcast, SVC, etc.) that are not included in WebCodecs. As a result, I can see how it might be desirable to be able to utilize Insertable Streams along with WebTransport for a scenario involving media upload (e.g. flow of media from browser client to server). For this scenario to work properly, it would be necessary for the bandwidth estimate from WebTransport to be propagated upstream to PeerConnection congestion control, so that the WebRTC encoder could encode at a bitrate appropriate for the network conditions encountered by WebTransport.

However, I am not sure where WebRTC fits into a scenario involving media streaming from server to client. Using WebTransport with datagrams as well as resilience (RTX, FEC) implemented in WASM it should be possible to transport containerized media produced by the server down to the client, which can then be fed to MSE. If there is a desire to avoid containerization, then instead of MSE the encoded media can be packetized/de-packetized in WASM and then fed to WebCodecs which can decode the bitstream. Given the MSE and WebCodecs options on the receiver, it isn't clear to me what additional value would be provided by WebRTC Insertable Streams.

[jianjunz]

Hi aboba, thanks for your reply.

I agree that bandwidth estimation from WebTransport is important to WebRTC encoder. It is one thing could help to reduce the latency.

RTP packetizer/de-packetizer, RTX, FEC are also helpful. As we already have them implemented in WebRTC, I'm wondering does it make sense to defined some sort of interfaces and expose them to web applications? So web developers don't need to re-implement them in WASM.