Add an example sending an encoded stream with libdatachannel

[moonsikpark]

A considerable amount of libdatachannel's users are trying to send an h264 encoded stream over WebRTC.

It would be nice to create a copy-paste example simmilar to media-sender example that receive packet from encoders like libx264, libavcodec, etc... (but preferably libavcodec due to it's large user base) and send it over WebRTC.

I will try to come up with such an example if I succeed doing so in my own project.

[guest271314]

Can't you just send the data over a data channel?

[moonsikpark]

Decoders in browsers are so delicate that they need the video stream to be perfect with no wrong values (e.g. pts, dts). On top of that, browser decoders are hard to debug because they don't produce any errors, they just fail silently. This makes it especially hard to debug issues with live streams.

It would be nice to have a reference implementation, and this could, I think, lead to upstream changes because there seems to be some things that need to be ironed out in the upstream library.

2022년 5월 23일 (월) 오전 12:43, guest271314 @.***>님이 작성:

Can't you just send the data over a data channel?

— Reply to this email directly, view it on GitHub https://github.com/paullouisageneau/libdatachannel/issues/626#issuecomment-1133922952, or unsubscribe https://github.com/notifications/unsubscribe-auth/ALL2E6PR6GZMRFUERTN2HCTVLJI2XANCNFSM5VW7PT5Q . You are receiving this because you authored the thread.Message ID: @.***>

[guest271314]

Browser side, on Chromium or Chrome, you can receive the raw data stream by any means, libdatachannel; fetch(), etc., utilize MediaStreamTrackGenerator https://github.com/w3c/mediacapture-transform with WebCodecs H.264 https://www.w3.org/TR/webcodecs-avc-codec-registration/ decode the image to ImageBitmap or canvas.

[guest271314]

This https://github.com/mattdesl/mp4-h264 winds up being brought up re browsers re and H.264, see https://source.chromium.org/chromium/chromium/src/+/main:third_party/openh264/README.chromium. The basic pattern I describe is parsing the data client side, with or without a Web API, something like this for Firefox https://plnkr.co/edit/4Tb91b?preview and this for Chromium https://plnkr.co/edit/gCjYSt?preview. Good luck!

[moonsikpark]

Thanks! I'll definitely have a look into that.

[guest271314]

@moonsikpark See also https://github.com/chcunningham/wc-talk.

[paullouisageneau]

While you are right that such a setup can be a solution @guest271314, sending the data over a transport protocol not specifically designed for real-time media will result in higher latency and jitter in general. For instance, Data Channels are buffered and implement congestion control, which are features that make them unsuitable candidates for real-time streaming like video conferencing or gaming.

[guest271314]

@paullouisageneau Yes, source code can be verbose to achieve a given requirement https://twitter.com/juberti/status/1083445783196663808:

How many lines of code are in Google's WebRTC implementation (https://webrtc.googlesource.com/src)? As of the end of 2018, it consists of 1.21M lines of code (up from 1.08M in 2017); for a comparison, this is 3x as much code as the Space Shuttle software.

However, that amount of code is not necessary to achieve the expected result. This C++ Native Messaging host https://github.com/guest271314/captureSystemAudio/blob/master/native_messaging/capture_system_audio/capture_system_audio.cpp - is 24.9KB compiled - that streams real-time, raw PCM system and/or specific audio output as JSON that I then write to a MediaStreamTrackGenerator (MediaStreamTrack) that can be added to a WebRTC PeerConnection MediaStream (and that I encode to MP3 in parallel https://github.com/guest271314/captureSystemAudio/blob/534633071e1e8b8fe7d389bbf97a5e00ec4f1bcd/native_messaging/capture_system_audio/background.js#L244-L267). The same procedure can be used for images, or both images and audio.

I think all of the bells and whistles can be over-rated, or, wind up being expensive to maintain. I posted the link to encoding images as JSON and streaming the same to demonstrate the procedure to reproduce the expected result can be achieved using ordinary HTTP request https://github.com/guest271314/native-messaging-php with with Fetch and Streams API, and/or Native Messaging using JSON, etc. (some WebCodecs serializing/deserializing experiments https://github.com/guest271314/webcodecs), using far less code than WebRTC streams or data channels for transmission. I'm just posting alternatives to libraries and transmission means that I have found to work - by multiple and continuously testing and experimenting - with the least amount of code.

[moonsikpark]

@guest271314, thanks for introducing me to webcodecs. This beautiful standard is just perfect for my use case. This solved the task that I've been struggling to achieve for a month.

[moonsikpark]

Seems promising, will definitely have a look. Thanks!

2022년 5월 24일 (화) 오전 11:07, guest271314 @.***>님이 작성:

This https://github.com/mattdesl/mp4-h264 winds up being brought up re browsers re and H.264, see https://source.chromium.org/chromium/chromium/src/+/main:third_party/openh264/README.chromium. The basic pattern I describe is parsing the data client side, with or without a Web API, something like this for Firefox https://plnkr.co/edit/4Tb91b?preview and this for Chromium https://plnkr.co/edit/gCjYSt?preview. Good luck!

— Reply to this email directly, view it on GitHub https://github.com/paullouisageneau/libdatachannel/issues/626#issuecomment-1135320598, or unsubscribe https://github.com/notifications/unsubscribe-auth/ALL2E6IQLCDYWRQQ2MZUUADVLQ2XPANCNFSM5VW7PT5Q . You are receiving this because you authored the thread.Message ID: @.***>