finos / SymphonyMediaBridge

The Symphony Media Bridge (SMB) is a media server application that handles audio, video and screen sharing media streams in an RTC conference system.
Apache License 2.0
54 stars 18 forks source link

RTC-14823: Video thumbnails are frozen when someone is screensharing #357

Closed RicardoMDomingues closed 4 months ago

RicardoMDomingues commented 4 months ago

We greedy allocate bandwidth to send screensharing which causes to drop camera video frames due to not enough bandwidth. This fixes the problem by trying to "borrow" bandwidth reservation from screensharing to send regular camera limits.

To avoid degradate too much the screensharing in favor of the other camera video, a new setting was added "slides.borrowBandwidthThreshold", where we will borrow the bandwidth only when the bandwidth is higher than the configured threshold (default 2500Kbits/s), that means, we we going to prefer dropping camera video streams when the available bandwidth is below "slides.borrowBandwidthThreshold". When it's higher, we are going to prefer to limit the slides bitrate and used some of the available bandwidth to send the video camera.

Also after calculate how much bandwidth we need to borrow for sending low quality video layer for the video cameras, we add to give some margin to allow some flotation on bandwidth estimation, so we don't disable the video cameras only because we have estimated a slight lower uplink.

Future improvements 2500Kbits/s it's seems to be good enough for screensharing FHD with low fps, but perhaps for 4K it would be better have a bit higher threshold. Other weakness of this algorithm is if we have a sudden a big reduction of uplink estimation, it could cause to drop the camera video screen for a moment until calculate a new value that we can borrow from the screensharing, even if the screensharing is not actually using all the allocated bandwidth. As SMB does not the resolution nor the fps, we could track the actually bandwidth the sceenshare takes and make the allocation based on that, so we could take better decision about when to drop the other videos