[1.x] Weird retransmission and framerate to 0 on recent chrome

[shmruin]

What version of Janus is this happening on? Janus 1.1.1, tested on chrome 108 ~ 111(canary).

Have you tested a more recent version of Janus too? Yes.

Was this working before? Yes, it was working on chrome 105 and before. This fails only after chrome 106 version and later.

Is there a gdb or libasan trace of the issue? No.

Additional context Chrome BWE looks working different on recent version(108 ~ 111).

When it was chrome 105 version, BWE with janus works fine.

I used Video "configure" with simple bitrate, and even on quite high bitrate (like 15M and more) BWE works good and no retransmission occur.

But recent release version of 108 and 111(dev), BWE makes retransmission bitrate and looks like AvailableSendBitrate is just fixed.

Also, Video is freezed (which is not on chrome 105), and framerate goes to zero.

You can see my test result - with bitrate configure 25M - here (https://codesandbox.io/s/beautiful-sky-7s1lqs?file=/index.html)

This results in a big problem for me because I can't use the same video bandwidth like before.

Any advice can be very helpful.

Thanks.

[lminiero]

I just tried the EchoTest and VideoRoom demos we have online with Chrome unstable on Linux (v110) and tinkering with the bandwidth controls seems to work as expected.

Edit: I see you mention 111 for Canary, which probably means 110 is not affected. I'll have to wait until Chrome is updated on Linux, as otherwise I don't have a way to test this.

[lminiero]

I did test Chrome stable, though (v108) and that one seems to be working fine too. I only played with the bandwidth control, though, which forces a limitation via REMB; so not really the BWE in action because of actual network constraints.

[tmatth]

We'd previously seen something similar (reported here) which we'd since avoided by:

• using a larger interval between PLIs (too short seemed to exacerbate the problem)
• trying to recover client-side with setLocalDescription() (no argument) followed by setRemoteDescription(pc.currentRemoteDescription) and escalating to an ICE restart if the FPS had still flat-lined

Neither of these are great options as they seem to be working around some unexpected Chrome+Janus interaction, but AFAICT the problem had to be on Chromium's side.

[shmruin]

Thanks for the reply. I set up the setLocalDescription with setRemoteDesciption as @tmatth mentioned, but this problem still occur. (But this time at least chrome try to recover the framerate, see https://codesandbox.io/s/funny-lederberg-4dq48m?file=/index.html) It really seems like chrome problem, but actually video is not freezed on bandwidth 15mbps with SDP (B=AS) settings without janus. You can test this here(https://codesandbox.io/s/bold-breeze-31700x?file=/index.html) on any chrome version. So far the only difference I noticed was that AvailableOutgoingBitrate on out-rtp. Old chrome slowly increase this value to the value of Videoroom configure bitrate, but recent chrome just fix this value to the configure bitrate at first (then no increase or decrease). Also, I set video_bitrate_cap of video room to true.

[lminiero]

If you're aiming for so high bitrates, just set bitrate=0 in the room and let the BWE do its job on its own. The bitrate property in the room uses REMB, so it may be REMB and BWE are fighting in your case.

[atoppi]

I have replicated the issue (Chrome/Linux 108). 1) Chrome seems to use maxBitrate as a "target" bitrate: as the available bandwidth increases Chrome tries to use it. I'm not sure if this is an intended behavior or not, imho that bitrate is wasted for such webcam video, indeed when left to the BWE Chrome stays around 2mbps. 2) When approaching 8-10mbps, even on local p2p local tests, Chrome starts experiencing packet loss. 3) Firefox does not have the same behavior, even setting 15Mbps as maxBitrate, the output bitrate sticks to 2-2.5 mbps. 4) When Janus is used as SFU, the server detects the packet loss on the publisher connection as expected but something is happening on the subscriber leg

Got 5 SRTP/SRTCP errors in the last few seconds (last error: srtp_err_status_replay_old)

My guess is that rtx over the publishing leg is leading to "too old" packets being relayed on the subscribing leg, due to very very high packet rate. On a pcap I noticed that rtx on the subscribing leg some packets have seq nums 100-200 lower than current seq nums, I would not be surprised if libsrtp discards them as replay packets.

[lminiero]

On a pcap I noticed that rtx on the subscribing leg have seq nums 100-200 lower than current seq nums, I would not be surprised if libsrtp discards them as replay packets.

rtx packets use a different SSRC, which means a different sequence number space (and different context in libsrtp). The "actual" sequence number is kept at the beginning of shifted payload in order to be restored by the recipient.

[atoppi]

On a pcap I noticed that rtx on the subscribing leg have seq nums 100-200 lower than current seq nums, I would not be surprised if libsrtp discards them as replay packets.

rtx packets use a different SSRC, which means a different sequence number space (and different context in libsrtp). The "actual" sequence number is kept at the beginning of shifted payload in order to be restored by the recipient.

my bad sorry for the confusion, I meant packets rtx over the publisher leg (so different ssrc) that get relayed over the subscriber leg (same media ssrc).

[shmruin]

As this issue is not replayed on chrome 105, I tried to check into the chrome commits of 106 version on webrtc. Looks like webrtc release on 106 is branch 5249, after commit dc5cf31c. And it seems there're some commits about bandwidth and estimations, like preference about high bandwidth or setting the max bitrate restrictions, and so on. Can these commits affect on the way janus working...?

[fippo]

if you can replicate, running bisect-builds.py might be the fastest way to figure out what change caused this exactly. There have been changes to packet pacing as well as BWE which makes this a bit hard to pinpoint

[shmruin]

It seems chrome accept and commit some bugfix that I registered as an issue related to this problem. I'm gonna test if this fixed in my local chorme.

[fippo]

That is unlikely to be the root cause since the send buffer has been "low" for a while and this seems to have worked previously

[shmruin]

Test result with chrome bugfix A little bit better than current chrome release but video is still freezed. Looks like this is not the root cause.

[lminiero]

Any update on this? If it can't be confirmed it's an issue in our code, I'll just close this.