search

paullouisageneau / libdatachannel

C/C++ WebRTC network library featuring Data Channels, Media Transport, and WebSockets
https://libdatachannel.org/
Mozilla Public License 2.0
1.81k stars 365 forks source link

Depacketizer for Track #676

Open paullouisageneau opened 2 years ago

paullouisageneau commented 2 years ago

It would be useful to introduce a media handler to depacketize the incoming RTP stream, with a jitter buffer to reorder packets and pace reassembled frames.

Sean-Der commented 1 year ago

@paullouisageneau I am going to start working on this.

Do you have any suggestions/rough sketch of an API you would like? I can get a PR together this weekend.

thank you!

paullouisageneau commented 1 year ago

Thanks! I've no very precise API in mind, I think something like a generic media handler implementing the jitter buffer part, an another one specialized per codec implementing the depacketization part would make sense.

Sean-Der commented 1 year ago

Hey @paullouisageneau I don't know the best way to do this. Can I get some guidance?

I don't see a way I can use RtcpReceivingSession because it is a MediaHandler.

Should I wrote something that allows a MediaHandler to be used as a MediaHandlerElement ?

Then I can contribute a JitterBuffer that is a MediaHandlerElement and chain them.

paullouisageneau commented 1 year ago

@Sean-Der I've merged https://github.com/paullouisageneau/libdatachannel/pull/929 which refactors and drastically simplifies media handlers. It should solve this kind of issue, as now everything is a MediaHandlerand you can simply chain MediaHandler instances together. It keeps compatibility with user code as long as it doesn't implement a custom MediaHandler.

The methods are also simpler, taking a vector of messages that may be modified in-place by the handler.

flying1314 commented 12 months ago

Hey @paullouisageneau I don't know the best way to do this. Can I get some guidance?

I don't see a way I can use RtcpReceivingSession because it is a MediaHandler.

Should I wrote something that allows a MediaHandler to be used as a MediaHandlerElement ?

Then I can contribute a JitterBuffer that is a MediaHandlerElement and chain them.

What is the current status of WHEP?@Sean-Der

singpolyma commented 8 months ago

https://github.com/paullouisageneau/libdatachannel/blob/master/include/rtc/rtpdepacketizer.hpp is mostly implementing this now, right? One think I'm curious about, once depacketized is it possible to still know what codec is in use so I know where to send it next?

Sean-Der commented 8 months ago

@singpolyma check out my code here https://github.com/Sean-Der/obs-studio/commit/506ecb62f3c1a2dbe6ad02c6481208f3bf7082c6

I accept + decode + render in OBS

singpolyma commented 8 months ago

@Sean-Der That looks very helpful, thanks! It does look like you're only advertising support for opus though, right? So you can just assume the packets have opus in them. Whereas if you advertise both PCMU and opus one needs some way to know what's actually being send. In the RTP packets we can look at the format number and compare to the rtpmap, but post-depacketize I'm not sure where that data is.

I'm also curious, you're using onFrame instead of onMessage is that a bit more low-level?

singpolyma commented 7 months ago

Answered some of my own questions and opened https://github.com/paullouisageneau/libdatachannel/pull/1156 which I think is needed for codec detection.

Sean-Der commented 7 months ago

Sorry I missed this @singpolyma! I get so many GitHub notifications I tend to ignore them :/

I am on the discord instance, happy to chat anytime :)