Audio/video chat room

[lazerwalker]

Naive first implementation:

• Upon entering a special room, you're added to a videochat with everyone in there, using a WebRTC full-mesh topology
• Toggles to not send your video and/or not receive video from anyone else
• Toggle to receive captions
  • If anyone toggles this, every single client generates captions for their own speech, and sends the raw captions over p2p data channel to people who request it. Gets displayed in chat scroll.
• Toggle to do text-to-speech
  • Submits mp3 taken from tts-ing what you type in the text chat
• For v1, no max room size. Good thing to test.

[lazerwalker]

The technical underpinnings of spinning up a WebRTC full-mesh topology are in place (just call "broadcastPeerId", and all peers in the same room will attempt to connect to you)

What's needed:

• [x] When in a room with media chat, add a 'join' button that, when clicked, calls broadcastPeerId
• [x] If you receive a broadcastPeerId request, don't respond unless you're already in webRTC mode (we already have a state variable to track this, we're just not respecting it currently)
• [x] A toggle exists to disable sending your video/audio
• [x] Indicate when somebody else is talking
• [x] Don't play your own voice
• [x] Investigate why this doesn't work for Em on Edge
• [x] When the AV selectors are initially shown, make sure they correspond to the video feed you see
• [x] Before joining, build out our own selector to pick what audio/video inputs you want (Safari doesn't offer a selector, so it's important for us to build this out ourselves). the Media API won't show you pretty names for devices unless a stream is open, so this requires a more complex flow. Clicking "Join" asks for a media stream, then shows you yourself, asks what devices you want, and has audio/video initial start toggles. We remember this for next time. There's a "Connect" button there that actually calls broadcastPeerId.
• [x] Next to the join button, show how many people are in chat (need to build out a server-side presence monitor like we have for regular room presence)
• [x] CSS work
• [x] Going to your profile shouldn't kick you out of chat
• [x] A settings icon exists to let you change devices mid-videochat
• [x] Stop dealing with 'is someone speaking?' data when no longer in chat
• [x] Migrate input selector to a modal
• [x] Turning off your video should remove the single freeze-frame
• [x] Figure out why "join voice chat" often doesn't work
• [x] <Placeholder TODO list item. Everything above this is okay for an MVP, everything below isn't>
• [ ] Still render selector view if getUserMedia fails (context: I have an unusable device on my Windows PC. Failing to load tht meant I couldn't load the selector to pick the right device)
• [ ] Show your own volume levels
• [ ] Remember which AV devices you are using for the next time you enter an audio/video chat
• [ ] A toggle exists to disable sending others' video (This is complicated, see comment below)
• [ ] Toggle to receive captions
  • Send message to everyone who's connected that you want captions.
  • When you receive caption request from at least 1 person, use Azure Speech Recognition to get a transcription for everything you say. Broadcast that over webrtc data to that person
  • When you receive caption data, display it as a message that indicates it's a transcript
• [ ] Toggle text-to-speech
  • If enabled, when you send a text message, either: generate an mp3 and stream that data, generate an mp3 and data-send the URL, or tell each other client to use client-side TTS

[lazerwalker]

Some WIP work was done to let you mute others' streams, but it seems not viable. It's on the mute-others branch.

I tried removing tracks from the stream for a given peer, but WebRTC explicitly yelled at me that I couldn't re-add a removed track, I needed to enable/disable the track.

However, right now, there's a single universal stream/track across all peers, so if I enable/disable, it will do it globally for all peers you're sending data to.

Open question: I can clone() a MediaStream to make a copy. If that has no performance ramifications, we can create clones of the mediastream for every peer and this will work.

[lazerwalker]

Note to self: do text-to-speech and speech-to-text on Twitchstream.

[lazerwalker]

Videochat may not be fully working, but closing this for now as a historical artifact