search

clowdr-app / clowdr

Midspace (formerly Clowdr) is a completely open-source virtual conference platform. Host complex events with hundreds of authors and presenters. Midspace includes automated livestreams, video chat, text chat, randomised networking and much more.
https://www.midspace.app/
GNU Affero General Public License v3.0
60 stars 20 forks source link

Live Transcription failing in video chats #402

Open EdNutting opened 2 years ago

EdNutting commented 2 years ago
  1. We repeatedly hit the "max 25 concurrent streams" connection limit (somehow - probably a side effect of the other issues below)
  2. A connection times out after 15 seconds if no audio data is received
    • This possibly immediately triggers an attempt to reconnect if the person's mic is on even if they are still not making any noise. Likely to trigger rate limiting.
  3. The connection token is fetched when the camera is first switched on. But if their mic is switched on after, this won't necessarily cause a refetch of the token. The token expires if it isn't used within 5 minutes.
  4. The connection token isn't refetched if it expires or otherwise causes an error
  5. Connections needs to be created (and token possibly refetched) when the mic on/off status changes not just camera object change (the connection probably does recreate on mic toggle with the current code but the token won't be refetched)
  6. Connections should not be created unless the user is actually talking. If someone is sat silently (with mic on) then we shouldn't try to transcribe until they make noise (and we need to buffer that noise for pushing into the websocket once it is established)