search

matrix-org / waterfall

A cascading stream forwarding unit for scalable, distributed voice and video conferencing over Matrix
Apache License 2.0
97 stars 5 forks source link

Investigate why changing input devices in the frontend app causes connection to freeze #34

Closed SimonBrandner closed 1 year ago

daniel-abramov commented 1 year ago

@SimonBrandner, should we close this one? It seems like we've found a reason for it (the client did not send an SDP offer).

SimonBrandner commented 1 year ago

Should it send one? Shouldn't it use replaceTrack()? Perhaps my memory is failing

SimonBrandner commented 1 year ago

Yep, we do a replaceTrack(): https://github.com/matrix-org/matrix-js-sdk/blob/matthew/sfu/src/webrtc/call.ts#L1396

daniel-abramov commented 1 year ago

Hm-hm. Is there anything on the SFU side that we need to track to make it work? So far it's observed as a freeze on the other participants' side. We don't seem to get any errors on the SFU side though 🤔

SimonBrandner commented 1 year ago

Yeah, I really have no idea what causes this... - it needs investigation

daniel-abramov commented 1 year ago

So I've tried it again a couple of times today. In all cases, we don't get any events from the client, and we also don't get anything from Pion (so no callbacks from onTrack or something [though probably it should not fire indeed if we're just replacing a track]). We also don't get any errors on our goroutine that reads the data from the track. So on the server side we don't have any indication that something changed and we don't know what to do next.

Interesting enough, the logs in dev console on the client side are the same for EC Full Mesh and EC with SFU, but the later does not work. Perhaps we need to also investigate it deeper on a client side somehow, not sure.

SimonBrandner commented 1 year ago

Turns out the fact we were not cloning streams causes the tracks to be accidentally stopped somewhere

(fixed by https://github.com/matrix-org/matrix-js-sdk/pull/2423/commits/ae2b66ed81b3cd2310c14c532cbfc1646c0d63d0)