SFU Server: Multiple users connected simultaneously, implement view-only password, isolated sessions

[ehfd]

Apache Guacamole and noVNC provide access to multiple users simultaneously to one screen. They also have functionalities which allow users to set view-only passwords for view-only screen sharing. It is required to investigate the feasibility of this function for the Selkies project to be used more widely as a stable remote desktop solution.

Also: what about isolated sessions for multiple users, where throwaway desktops are created and scrapped for different users? Such environment probably requires a lot more things than just Selkies-GStreamer, but the scope for the remote desktop system is addressed here.

[ehfd]

Relevant code: https://github.com/selkies-project/selkies-gstreamer/blob/master/src/selkies_gstreamer/signalling_web.py Should be addressed with #27. The signalling server must be rewritten to accommodate multiple peers. Anyone with experience in WebRTC development can do it.

[ehfd]

This might be addressed with #48.

[nums]

To be able to send stream to others participants, I think the ideal way is to send stream to a WebRTC SFU server. Neko project is using Pion WebRTC (https://github.com/m1k1o/neko) to achieve that. Webrtcbin must be able to interface with SFUs (there is a WHIP client that uses webrtcbin https://github.com/meetecho/simple-whip-client) Once done, the main problem is the management of the quality of the streaming with the recovery of Nack events, PLI etc and this can produce lags or a noticeable drop in the quality of the video.

The mediasoup documentation explains this very well: https://mediasoup.org/documentation/v3/scalability/#one-to-many-broadcasting

[ehfd]

@nums Thank you for the information! We are currently focusing on improving the documentation for more contributor accessibility. We have a goal for this project to potentially attract community contributors from every relevant field, from experienced GStreamer/WebRTC/conferencing/media/graphics software developers to video power users who simply know a lot of about optimal encoder parameters, and Python/JS learners.

[ehfd]

https://github.com/centricular/webrtcsink instead of webrtcbin may solve this issue as this has its own signalling server.

[nums]

It can, but they incidate that :

It is important to note that at this moment, encoding is not shared between consumers. While this is not on the roadmap at the moment, nothing in the design prevents implementing this optimization.

So it is possible, but there will be a big use of resources. Some certain nvidia card have a limitation of number of encodings so it can be a problem.

I'm working on my side to use webrtcbin with a SFU with this "bridge" https://github.com/meetecho/simple-whip-client and to stream the desktop screen to an SFU like mediasoup. I just have a little issue on the compilation with the selkies gstreamer version, but "work in progress"

[ehfd]

@nums I guess that you are working on a contribution. We greatly thank your time.

[nums]

Yes I'm trying to share the screen in "readonly" mode, but to be able to take the control il will be an other story

[ehfd]

Yes I'm trying to share the screen in "readonly" mode, but to be able to take the control il will be an other story.

That's already something we really need. Thanks a lot!

[ehfd]

Relevant: https://github.com/selkies-project/selkies-gstreamer/issues/48#issuecomment-1687298232

[ehfd]

WebRTCSink is a wrapper for WebRTCBin, where settings are available in https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/blob/main/net/webrtc/src/webrtcsink/imp.rs

The issue of WHIP/WHEP is that there is no DataChannel for keys, cursors, and other information. Only read-only access would be possible without other transport protocols.

Thus, the immediate property might rather be a multi-user SFU server.

Need contribution for the SFU server, perhaps utilizing GStreamer's own signalling server.

https://gstreamer.freedesktop.org/documentation/rswebrtc/index.html#usage

[PMohanJ]

The current architecture of selkies-gstreamer consists of two webrtc connections(video and audio) for one single session to a user. So using plugin webrtcsink would result in 2*n connections (of course, all are handled by webrtcsink itself) for n number of session?

If so, considering the below note, wouldn't it be resource intense even for as little as 2-3 users?

It is important to note that at this moment, encoding is not shared between consumers. While this is not on the roadmap at the moment, nothing in the design prevents implementing this optimization.

[ehfd]

So using plugin webrtcsink would result in 2*n connections (of course, all are handled by webrtcsink itself) for n number of session?

This is called an MCU server.

An SFU server distributes the same stream(s) to many clients. I still think webrtcsink's signaling server is still capable of doing this.

https://forasoft.medium.com/p2p-vs-mcu-vs-sfu-what-to-choose-for-a-video-conference-app-fa5cb4a5dd6e