It's hard to tell when media links are slow

[karth295]

There are a couple of user stories here:

1. While my local stream appears to be near perfect, it's choppy/frozen to others. I keep talking confidently, without feedback that others can't really see/hear me. I want to be able to tell when others perceive me as choppy/frozen.
2. I see choppy video streams for others, and I don't know why it happens--maybe tawk.space is slow. I want feedback like the media link's bitrate to tell me what's going on.

[toomim]

I just realized we have three issues open for the same thing: https://github.com/invisible-college/tawk.space/issues/8, https://github.com/invisible-college/tawk.space/issues/11, and this one.

Next step: summarize them all into a single issue.

My current idea is to visualize how well other users can see you. Imagine their video feed displays an orange border when they are dropping packets of yours, and turns red if they can't see you at all.

To implement: each user will send a heartbeat state to each other user through the server. This heartbeat will report the most recent frame of video seen, and also how many dropped frames occurred in the last second or so. Then we can display a red border if the most recent frame is older than 2 seconds, and orange if dropped frames are > 10%.

[toomim]

This design is kind of ugly, but gets the point across:

Editable here: https://www.figma.com/file/SBDi2BDaGDELPP85trlILsBi/tawk

[Delta-neutral]

I kind of like the design

https://www.figma.com/file/SBDi2BDaGDELPP85trlILsBi/tawk

[karth295]

This does give the user a little more social translucence. But my theory for why other video chats haven't done this is that there's nothing actionable you as as a user if you can't see/hear somebody or vice versa.

Any thoughts?

[toomim]

Oh, lost information is very actionable!

If you make a point to the group that you want them to know, and they are highlighted as "can't hear you", then you can repeat your point!

If you notice that someone else didn't hear a speaker for a bit, then you can add redundancy when you reference their point that might not have been heard.

If you can see a pattern in someone not seeing or hearing people, then you can realize that your internet is slow, or that it happens when you have another tab open (which is hogging all your CPU) and you can actually fix the hardware that is causing the problem! The first step to solving these problems is to see when they are happening, so that you can notice correlations with when they happen, and fiddle with the things that correlate.

[toomim]

Karthik describes how to get info on bad streams here:

they come from Janus. I really dislike their API -- you get an "onMessage" callback invoked and it can be one of several messages. The "slow_link" message that we use indicates the other party sent a bunch of NACKs in a row.

https://github.com/invisible-college/tawk.space/blob/master/index.html#L582 https://github.com/invisible-college/tawk.space/blob/master/index.html#L668

One of those refers to you uploading your stream to the server, and other is downloading another person's stream.

[karth295]

From my friend Kyle offline: just give people a button to click saying they cannot see or hear someone.

[toomim]

People do the equivalent of that right now by saying out loud "I can't see/hear you", or typing it into the scratch area.

[karth295]

Eh: 1) If you can't see/hear someone they probably can't see/hear you complaining 2) Scratch space isn't as visual/obvious

I see buttons at least as a way to easily prototype the UI we're proposing.

[toomim]

We also text each other via sms, imessage, email, etc. This works well enough for the course-grained feedback on being able to hear eachother, like to communicate "the connection is generally bad", or "I can't see you at all."

The advantage of showing it from network traffic is that you'll get precise, fine-grained feedback, and it'll be reliable. You could know exactly which words, or which part of your sentence you need to repeat. You'll know whether the problem is due to your internet, or due to the recipients network. You'll know whether there actually is a problem, or whether the person on the other end clicked the button because they didn't know what it does, or whether they neglected to click the button, because their mouse cursor was in a different window, or a different tab, or because they don't know about the button, or because they are lazy.

[karth295]

Fair enough -- I agree with you long term. However, I'm not yet convinced that we can reliably detect poor connections. I remember I made tawk pop up a notification on every "slow_link" message, and it was spamming everybody with false positives.

I think step 1 is to figure out whether the red/orange boxes are even helpful/actionable information. And we can do that quickly by adding buttons and testing with people who aren't lazy.

[toomim]

The problem with those messages was that they were popup notifications. Their information would be great presented as an ambient display -- like a large dot that changes color in the corner of your screen, or pulses with each lost message, or an hourglass icon that appears, or perhaps a wifi icon that goes grey, or has a line through it, or just loses bars. The idea is to make the information available, but not to demand the user's attention to it. The user can attend to the info when they need, or can keep an eye on it. Notifications were especially annoying back then because they made sounds.

We know that this information is helpful and actionable, because people already ask "Can you hear or see me?" periodically, and then take actions based on the answer to that question. I probably hear an average of 5 "can you see or hear me?" questions per week in tawk.space.

We described actions one can take in an earlier post:

• Repeat your point
• Talk more slowly, or using redundant word or expression when your link is bad
• Change wifi networks, or reboot your modem, or switch to LTE, or buy a new computer, or cancel comcast.
  • Oh wait, you can't cancel comcast, beacause it's a monopoly.

[karth295]

Making notes to myself: Janus does have a better interface for counting NACKs -- there's a callback slowLink(is_remote_stream, nacks_in_the_last_second). E.g. https://github.com/meetecho/janus-gateway/blob/master/html/nosiptest.js#L148.

[toomim]

Good!

I don't understand what "remote stream" means, or what a "link" is. I'm guessing that, in a 4-person video chat, there might be 4 streams accessible from each client:

• 1 stream uploading from the client
• 3 streams downloading from the server, one for each person

Maybe the latter three of these are called remote streams, and the upload is called a "local" stream?

(I'm assuming that all this code is on the client. It would be cool to know if we could get network speed info from the server's perspective too!)

[karth295]

Yes, remote stream is a stream you (the client) are downloading. Link in this case is referring to a stream between two computers, rather than a physical cable. You get a slowLink callback if the receiver has been sending NACKs. For a local stream, you get the slowLink callback when Janus sends you NACKs, and for remote streams when you send Janus NACKs.

So "you cannot see X" means either you are giving Janus NACKs for stream X, or Janus is giving X NACKs when it is trying to upload. Similarly, X can't see you means Janus is giving you NACKs when you're trying to upload your local stream or X is giving Janus NACKs.

Other things we can look at:

• Bitrate for remote streams (may help us understand quality)
• Receiver Estimated Bandwidth (though I don't know how to get this info from Janus: https://webrtcglossary.com/remb/

[toomim]

Thank you, this is useful information!

Can we add hooks on the Janus server every time it sends a NACK to a client? That would be the most actionable information—that your video isn't reaching the server, and thus isn't reaching anyone. If we can write a function that gets triggered when the Janus server notices it, then we could make that function update the tawk.space server's state, to visualize the source of the bad connection in everyone's client.

[karth295]

Janus is split into a core, which implements webrtc, and "plugins", which add functionality like rooms and recording. We use the videoroom plugin. One of the plugin callbacks is slow_link (some number of NACKs in one second): https://github.com/meetecho/janus-gateway/blob/master/plugins/janus_videoroom.c#L3370. We could in theory modify this callback to inform statebus, or just write a new plugin entirely that's integrated with statebus.

But that seems too complicated for now. We can already mimick this behavior. I assume clients are mainly in one of two scenarios: 1) A client has at least some internet connection, so can tell statebus link statuses, and get other people's link statuses from statebus 2) A client doesn't have an internet connection, and statebus doesn't get heartbeats, so statebus assumes they are not able to upload video. The client isn't getting heartbeats from statebus, so it can assume it can't hear anybody. We can make heartbeats pretty frequent (let's say every 2 seconds with a 1 second timeout)

[toomim]

Thanks Karthik! This slow_link handler in the videoroom plugin is looking really tempting to me. It will trigger whenever there are more than 8 NACKs per second.

I'm considering modifying the code in janus core (it's probably no harder to modify than the plugin) to output the number of NACKs directly onto statebus. We'd modify this function here: https://github.com/meetecho/janus-gateway/blob/95d0ba3ef9757cca7908255b814e1091c1423e8c/ice.c#L1448

I think this is the opposite of the direction you're interested in—to approximate the video network loss by watching for websocket network loss—but if we look at the actual packets lost, our information becomes so much more precise. Precise information makes debugging so much easier that it's really attractive to me.

I'm certainly not asking you to implement it, but I'm very grateful for your help in locating the place where the work needs to be done!

[karth295]

Yeah, absolutely. I think you'll still need deal to deal with the case where a client gets disconnected from the network and cannot send NACKs to Janus to trigger that function. Tawk should also probably tell that client that they're offline.

[toomim]

Oh, shoot. What we want isn't the NACKs being sent by the client, but the NACKs being sent by Janus. We want to know when Janus is sending a NACK. That means that the client isn't sending packets, which means that nobody can see that client.

[karth295]

That's not all we want, necessarily. I also want feedback that another user can't see me because their internet is congested.

[toomim]

Ok, that's a good point. Thanks.

So the ideal setup (to me) tracks, on the server:

1. When Janus sends NACKs to a client
2. When a client sends NACKs to Janus

And Janus responds, in each case respectively, by setting statebus state that displays for every client:

1. Can't speak/show (their internet uplink sucks)
2. Can't see/hear (their downlink sucks)

Thanks again for your help thinking this through, the design is getting clearer and clearer to me.

This would have helped the other day when I was talking to Paul, and he could hear/see me perfectly but his uplink was too slow and I couldn't see/hear from him. He had no idea that his speech was choppy, until I told him, which surprised him because he could see/hear me so well, which probably led him to think that his internet connection was working great. (And I still don't have proof that it was his internet, and not mine—just a strong intuition.)

[toomim]

Oh, and I still don't totally understand the issue you raised (4 messages ago) about a client getting disconnected from the network. What's the situation? Do you mean disconnected from Janus, but still connected to the tawk.space statebus websocket? Or what?

[karth295]

Let's say there are only two users, X and Y. You need to think about how to capture and present the cross products of states that X and Y can be in.

X is fine	X upload is slow	X downloading Y is slow	X is disconnected
Y is fine
Y upload is slow
Y downloading X is slow
Y is disconnected

Most of these cases are valid scenarios. How does the UI look for each of them in each scenario? How can both of them receive the adequate information they need to show the UI you want?

Side note: I'm not sure if "Y upload is slow" implies "X downloading Y is slow".

[toomim]

This is mostly addressed by the UI proposal at the top of this thread. We visualize on top of each user whether or not they are able to speak/show, and whether they are able to see/hear.

In other words, we don't need the full cross-product, because the visualization of X and Y are independent. Each client (X, Y, Z, or Henry) gets their own little indicator of how well their internet is doing, irrespective of others.

However, a problem with those mockups is that they distinguish between "speaking" vs. "hearing". You've helped me realize that it makes a lot more sense to distinguish upload vs. download instead, which maps to "speak/show" vs. "see/hear".

[toomim]

Also, those mockups are about who can see/hear you, but after this conversation, I think we want to ignore the "you" part and just say "can't see/hear" or "can't broadcast" in general, as measured by the server.

[toomim]

I'm using https://meet.jit.si right now, and check it out — you can see how strong people's network connections are!

You can see at the bottom-left, a little popup shows network status when you mouse over a person's icon. Also each person's icon changes depending on their network quality.

Here's a waning network link:

And here's a link that's recovered and is good again:

This person has gone cold entirely:

When I mouse over their icon, it says "inactive".