Support data channels

[mat-hek]

Having data channels will enable membrane users to send custom data over WebRTC, but it will also let us avoid using signalling for sending some media-related data, like marking active speakers.

This may require improvements in https://github.com/membraneframework/ex_dtls and support for SCTP protocol.

[madawei2699]

Hi @mat-hek I want to implement the webRTC datachannel with the SCTP protocol, as I know, maybe needs these tasks:

• Create a membrane plugin like membrane_sctp_plugin, and this can wrap the erlang/otp gen_sctp to implement.
• Modify the ex_dtls to support the SCTP protocol.
• Modify the membrane_webrtc_plugin to support the webRTC datachannel.
  • The datachannel use ex_dtls to transport the data stream from the client.
  • Modify the membrane-webrtc-js to support the webRTC datachannel.
  • Modify the membrane_rtc_engine to support the webRTC datachannel.

Is that so? I would appreciate if you could give any help.

[mickel8]

Hi @madawei2699, sounds good but it would be nice to think more about passing messages from one data channel to another inside SFU. At the moment, engine supports publishing audio and video tracks only, data channels would require some changes in this API.

[madawei2699]

Hi @madawei2699,

sounds good but it would be nice to think more about passing messages from one data channel to another inside SFU. At the moment, engine supports publishing audio and video tracks only, data channels would require some changes in this API.

Hi @mickel8 Thanks for your advice. It seems one feature needs to change many repos, so maybe better to use Monorepo to organize the codebase? How do you think?

[mat-hek]

@mickel8 indeed, but I think that can be the last part. Hi @madawei2699, glad you'd like to contribute that. Regarding DTLS, it's going to be: ICE <-> DTLS <-> SCTP <-> ... That means:

• ex_dtls may not need to support SCTP specifically, it will just encrypt/decrypt and forward packets, so it may be already ready to go. Check RFC8831, especially section 5, for details. However, the ICE plugin only uses ex_dtls to perform the DTLS-SRTP handshake, so we'll need to figure out how to support DTLS encryption too (move DTLS to a separate Membrane filter?)
• SCTP should exchange data with DTLS instead of opening its own socket. This may make it impossible to use gen_sctp, unfortunately (to be verified)

Apart from RFC, this may be helpful: https://webrtcforthecurious.com/docs/07-data-communication/ Feel free to ask if you have any doubts ;)

[madawei2699]

@mat-hek Thanks for giving so deeply details. Seems to support the webRTC data channel, we need to do these tasks:

• Creat a SCTP elixir version like ex_libsctp, maybe can refer the golang version like sctp, and this lib need support ex_dtls.
• Create a membrane plugin like membrane_sctp_plugin, and this can wrap the ex_libsctp.
• Modify the membrane_ice_plugin to support the SCTP and SRTP protocol, which need to figure out how to use DTLS to encryption SCTP and SRTP packages.
• Modify the membrane_webrtc_plugin to support the webRTC datachannel. The data channel use ex_dtls to transport the data stream from the client.
• Modify the membrane-webrtc-js to support the webRTC datachannel.
• Modify the membrane_rtc_engine to support the webRTC data channel, need consider how to pass messages from one data channel to another in the engine.

Any comments?

[mat-hek]

Almost. Here is how I see it, the top pipeline shows how it's now, and the bottom is my view of how it's going to be:

So we need

• ex_libsctp that would handle exchanging packets that are Elixir binaries
• membrane_sctp_plugin as you wrote, with something like Membrane.SCTP.Filter
• new Membrane.DTLS.Filter that would
  • handle DTLS-SRTP handshake
  • decrypt inbound DTLS packets and forward them to Membrane.SCTP.Filter
  • forward inbound SRTP packets to Membrane.SRTP.Decryptor (these are not DTLS packets, so just forward them, SRTP is responsible for decryption)
  • encrypt outbound SCTP packets and forward them to ICE
  • forward outbound SRTP packets to ICE (again, no encryption needed) (all this should be possible with current ex_dtls)
• then go webrtc_plugin, webrtc-js and rtc_engine

In the webrtc_plugin you'll need to negotiate data channels in SDP, which is quite tricky. I'd probably start from there to actually get a data channel from a browser. Once you have that, try to decrypt what you receive using DTLS and finally handle SCTP. Then the other way round ;)

[madawei2699]

Thanks @mat-hek Now it is more clear to me. I can try start to implement the ex_libsctp, do you think possible to call some c SCTP library or must need to implement full protocol by pure elixir?

[mat-hek]

I think it's best to have a POC with minimal effort to validate objectives. So I'd rather use something ready if possible, at least for now. I'd also check the Erlang implementation, maybe ask at the erlang forum - maybe there's a way to use it.

[madawei2699]

Ok, that make sense, I will try this approach first, and will post updates in here.

[bugnano]

another huge advantage of supporting data channels would be that it will be possible to use Membrane to create a real-time game server. For example the game is a client that uses https://github.com/geckosio/geckos.io , and the server can be implemented in Elixir instead of JS, which would be a huge plus