matrix-org / waterfall

A cascading stream forwarding unit for scalable, distributed voice and video conferencing over Matrix
Apache License 2.0
97 stars 5 forks source link

Detect stalled publishers (layers) and switch to other publishers (layers) #147

Closed daniel-abramov closed 1 year ago

daniel-abramov commented 1 year ago

When simulcast is used, we get different publishers (a single publisher per simulcast track/layer). The browser may decide to stop sending RTPs to specific publishers, so we will stop getting RTP packets.

If there is a subscriber X that is subscribed to publisher Y and publisher Y does not emit any frames, subscriber X will observer it as a "video freeze" (if it's in the middle of the conference) or as a "grey frame" (if it's at the beginning of the conference).

We solve the issue by detecting such stalled publishers Y. Then we gather the set of subscribers that are subscribed to the given publisher and let them receive RTP frames from a different publisher Z (as long as the publisher belongs to the same track), i.e. effectively we detect when e.g. 3 users are subscribed to the high-quality video of a user Y and then we notice that user Y stops sending the high quality, but continues to send the low quality, we switch all these subscribers to the low quality.