Mesh network group video calling

[aryavohra]

We've added support for peer-to-peer group video calls for 4 peers, as requested in the linked issue. This is implemented by using multiple RTCPeerConnections per VideoChat client.

The group call actually works, and is done! We are currently just making some small fixes / improvements.

@ianramzy we'd be happy to help load test this and figure out what an appropriate upper bound for group call size should be, although we bet 4 is a good limit.

CSS fixes

• [x] Assigning each user a random color for use as border around user's video and displaying in chat
• [ ] Scaling for screensharing
• [ ] Video tile wrapping for 3, 4, 5 peers
• [ ] Adjust video tile size for 2 peer calls
• [ ] Wrapping tiles for mobile

Logic fixes

• [x] Mute mic
• [x] Fix picture-in-picture
• [ ] Experiment with downscaling stream quality with 4/5 peers
• [x] Distinguish between network disconnects and peer leaving room
• [x] Disabling local video stops peers from connecting
• [x] Fix peer joining when screensharing
• [ ] Add check so that it doesn't crash if you send a message when someone is joining

Testing

• [ ] Check for random refreshes
• [ ] Load testing

By: @taixhi @khushjammu @aryavohra

[ianramzy]

Yo this is is awesome, I am super busy this week so no promises I can get this in quickly. But this looks awesome at a hight level (will look into code later). Two things I noticed:

1. I am not a fan of the nicknames I would like to see that removed. If that is being used for some sort of UUID hashing then just assign new users a random string and use that. I want to keep user friction as low as possible.

2. CSS for incoming video feeds.

   • The video tiles don't wrap properly for 3,4,5 people in the call. I have to zoom out to 90% for them to wrap properly onto a 2x2 grid. Should be a small amount of CSS :)
   • When in a 1 on 1 call the received video feed of the other person is too small, it should be close to 70% the size of the screen. Similar to what it was before this PR on the current master. Again, should be a small amount of CSS :)
   • When I screenshare the video tile becomes too large and wont fit on the recipients feed. This is again more CSS. It would be ideal if When someone is screensharing they're portion of the screen increases. But for now if it is split evenly that works aswell.
   • I think a lot of this is caused by scaling the video relative to its incoming resolution. I am sure there is a grid system out there that works

This is such awesome work guys, I am so happy to see this! I cant wait to get this in! Great job so far! 🚀

[aryavohra]

Thank you so much for the detailed feedback!! We'll hop on and make these changes ASAP and will get back to you once we're ready for review again.

We added in the nicknames so when using messages with >2 peers, users can keep track of the conversation. We could use friendly uuids instead though, e.g Blue Buffalo (adjective + animal name).

[ianramzy]

I think not showing nicknames at all will fine in the text chat.

[aryavohra]

Got it, will remove them. Thanks :)

[ianramzy]

5 person also barely works on my iPhone 6s, this is just a limitation of peer mesh and the high level of compute required to decode 4 incoming videos. I would like to experiment with dropping the resolution and bitrate of videos as the number of callers increases.

[ianramzy]

Also, dont worry about captions, I am going to remove them in the future. We will save removing them for a different PR however.

[aryavohra]

5 person also barely works on my iPhone 6s, this is just a limitation of peer mesh and the high level of compute required to decode 4 incoming videos. I would like to experiment with dropping the resolution and bitrate of videos as the number of callers increases.

We can give this a shot too, adding to todos

[Chaphasilor]

I just checked it out, great work! :D

The JavaScript seems to be working fine for the most part, including graceful disconnects, etc. However, the mute button isn't working for some reason.

Also, there are some other issues, especially on mobile. For example:

• [ ] the things @ianramzy listed
• [ ] on mobile, if more than 2 people are connected, the body becomes larger than the viewport. This results in the user having to zoom out and the UI getting messy.
• [ ] the nicknames are an interesting idea, but the UX isn't great with the prompt() part. I'd recommend assigning each user a random color instead. The color could then be shown as a frame around the user's video and also be shown in the chat :)

Really cool of you guy to make this PR though! <3

[aryavohra]

Love the color idea, adding it in to the todos

[ianramzy]

Make sure that there are no duplicate colored borders, will most likely have to persist a map of whos color is whos.

[Chaphasilor]

Make sure that there are no duplicate colored borders, will most likely have to persist a map of whos color is whos.

Actually, the colors don't have to be the same for each client: they just have to be unique for each client. So, just ID each client and distribute the colors in the frontend ¯\_(ツ)_/¯

Edit: wow, reading this again I made no sense at all 😂

What I meant is: Given that on client A, client B's color is green. This doesn't mean that B's color is green for client C as well. Every client assigns a (for them) unique color to all of the other clients, but they don't have to sync the color palette with the others. Does that make more sense?

[aryavohra]

So we decided to do colors that match across all clients. We're using UUIDs to deterministically generate unique colors so we don't have to send any color data back and forth, check it out!

[aryavohra]

We're fixing picture-in-picture and we're not sure how to decide which remote peer to show in the pop-out window. We're thinking we could ask the clients which peer they want in picture-in-picture.

Ideally, we'd dynamically change the picture-in-picture video based on who's talking, but I that's better suited for a separate pull request.

[Chaphasilor]

We're thinking we could ask the clients which peer they want in picture-in-picture.

Don't ask the user. PiP is not that important.
Just auto-advance to the next client after a fixed amount of time, like 5 seconds :)

[ianramzy]

For Pip I wonder if there is some way to group all the videos together, I doubt it, but for now asking would be ideal, picking last joined client will also work.

[Chaphasilor]

For Pip I wonder if there is some way to group all the videos together, I doubt it, but for now asking would be ideal, picking last joined client will also work.

I disagree about the 'asking the user' part. It's bad UX (just like asking for a nickname) and will cause headaches if the user wants to switch to showing a different peer at some point.
Last joined might work, but isn't ideal nor the expected behaviour. I'd only fall back to this if the auto-advance is too hard to implement.

[ianramzy]

I disagree about the 'asking the user' part. It's bad UX (just like asking for a nickname) and will cause headaches if the user wants to switch to showing a different peer at some point. Last joined might work, but isn't ideal nor the expected behaviour. I'd only fall back to this if the auto-advance is too hard to implement.

Maybe then we can monitor the audio levels somehow and dynamically change the video input on a special PIP video.

[aryavohra]

Found a pretty lightweight package for detecting who's speaking — what do you think? https://github.com/otalk/hark

[ianramzy]

I see, could be interesting, then the next question is can you hotswap video src in PIP.

I would also focus on fixing the video grid layout and the stability, those are top priority right now. I have been testing groupcalling and having mixed results. Hard to pinpoint what is happening exactly.

[aryavohra]

I see, could be interesting, then the next question is can you hotswap video src in PIP.

I would also focus on fixing the video grid layout and the stability, those are top priority right now. I have been testing groupcalling and having mixed results. Hard to pinpoint what is happening exactly.

Gotcha, we'll prioritise finishing up the CSS. Could you elaborate a little on what issues you're having with the group call itself — dropped frames etc? We have only been able to get smooth performance on group sizes of 3 & 4, beyond that we've had trouble too.

[ianramzy]

Gotcha, we'll prioritise finishing up the CSS. Could you elaborate a little on what issues you're having with the group call itself — dropped frames etc? We have only been able to get smooth performance on group sizes of 3 & 4, beyond that we've had trouble too.

4 person is about the max I have had success with aswell. The main issues I have experienced are:

1. When people disconnect for whatever reason it doesnt seem to reconnect / rejoin until a manual refresh happens
2. When a new caller joins sometimes they will only connect to some of the people in the call but not all of them.

[aryavohra]

Added in the downscaling for n>4 nodes in call — I set it to downscale to a fixed lower resolution of 360p. Not quite sure how we should rescale when the number of peers drops back down.

1. When people disconnect for whatever reason it doesnt seem to reconnect / rejoin until a manual refresh happens

I'll add in auto refresh again to solve this, we removed it but forgot to add it back lol.

2. When a new caller joins sometimes they will only connect to some of the people in the call but not all of them.

Haven't seen this yet myself, what are the steps to recreate?

Edit: figured out how to recreate issue 2. Seems like the peer discovery is unstable with larger rooms, working on fixing this and will hopefully be able to push a fix alongside all the CSS fixes today.

[aryavohra]

BTW, we were wondering if you guys had test checklists / unit tests for before pushing to master, we want to catch more errors before we push them. Thanks

[Chaphasilor]

For now there are no automatic tests in place. We'll try to manually test as much as possible ¯\_(ツ)_/¯

Any ideas how we could implement automatic testing?

[ianramzy]

Couple of thoughts:

1. I have a checklist for features to test, but its outdated so I wont include it. Making one would be a great idea. And automated testing would be even better.
2. Since we only use atmost 5 colors some of them are very similar and hard to distinguish, I like the choice of pallete, they just need to be more different.
3. Scaling down to 360p helps a lot, 4 person calling is almost workable on my iPhone 6s now, perhaps dropping the bitrate even more could fix that, will keep testing.

This is great work guys, I will make sure you are recognized in the Readme!

[khushjammu]

Thanks Ian, we appreciate it! We've added a few changes

• Stability has been improved. We tested calling a friend in the Philippines who doesn't have great wifi — it reestablishes the peerconnection smoothly. Achieved by making more intelligent use of the signalling server; for example, we use the signalling server to indicate whether a peer disconnection is intermittent (i.e. will be restored momentarily) or permanent. More generally, you shouldn't see any random failures like there used to be.
• Colors are working great. @aryavohra created a neat algorithm to deterministically calculate what each peer's color should be using just their UUID. This means all the color handling is handled on the frontend, and its consistent across peers.
• Peer joining during screensharing works great now. It used to crash or not have audio previously, but that's been fixed now.

The big one left is CSS. After that, we should be ready for more rigorous testing, and eventually merging it! Downscaling hurt stability, but we'll take a look at adding it back in.

BTW, we'll try coming up with a checklist and sharing that when it's done too.

[ianramzy]

Just tested it out with a group of 4 and while I could see everyone in the call, none of the other participants were able to see each person. I think the connecting stability still needs some refinement, perhaps it was because we all joined at the same time. Also downscaling is really needed for calls of more than two.

[Chaphasilor]

I thought downscaling to 360p was already enabled? @ianramzy are you talking about lowering it even more?

[ianramzy]

I thought downscaling to 360p was already enabled? @ianramzy are you talking about lowering it even more?

It was disabled for now I believe.

[khushjammu]

We're not sure how to replicate the issue. We tested it with five people on separate devices across networks and it worked fine. Could you please give more details?

It might be worth looking at deployment. We used Heroku to deploy @ meet.questo.ai — feel free to test it out with that and see if the issue persists.

[Chaphasilor]

@ianramzy how about we deploy this as a 'beta' feature, with some sort of disclaimer? I know you're not going to like the idea, but given that this is a feature that is hard to test and requires a lot of testing in general, we could add in some kind of feedback option? A simple star rating after the call ends, or a link to a short survey?

[aryavohra]

By the way, we'd be happy to schedule a call or something so we can hammer out the last few fixes and integration together :)

[Chaphasilor]

I think @ianramzy is pretty busy as always, but I'd also be down for a 'proof of concept' group call!

[Chaphasilor]

I'd say if @ianramzy doesn't respond soon, you should probably just customize your fork and offer it as an alternative to this project :)

I'd hate to see your work be in vain :/

[taixhi]

yeah it is a shame.. that's probably what we'll do.