High latency when audio is enabled.

[danisla]

Enabling audio results in high latency (700-1000ms) after first connection. It eventually drops down to something more reasonable (100-250ms).

[ehfd]

Thanks for all the information. The spec makes things very complicated. @wanjohiryan

[ehfd]

Perhaps, using good old WebSockets might not be that bad?

[alxlive]

The reported latency in the side panel is calculated by summing the latency components reported by the WebRTC stats. It's effectively network latency + video jitter buffer latency + audio jitter buffer latency. The code is here: https://github.com/selkies-project/selkies-gstreamer/blob/master/addons/gst-web/src/app.js#L378 This probably needs to be double-checked to make sure we aren't double-counting the audio and video latency, since with AV sync they may be overlapping. It might be that we need to take the max(audio, video jitter buffer latency) rather than adding each separately. I haven't confirmed this yet though.

I borrowed some of the code from your project and used it in ours to compare using 2 webrtcbins vs 1 with sync enabled/disabled (via do-timestamp). Here are the results for video after a few minutes of playing colorful animation scenes:

1. Audio & Video in one webrtcbin and sync enabled

{
  "connectionPacketsReceived": 98305,
  "connectionPacketsLost": 9134,
  "connectionBytesReceived": "144.38 MBytes",
  "connectionBytesSent": "0.66 MBytes",
  "connectionVideoLatency": 136,
  "connectionCodec": "VP9",
  "connectionVideoDecoder": "libvpx",
  "connectionResolution": "1280x720",
  "connectionFrameRate": 60
}

2. Audio & Video in one webrtcbin and sync disabled

{
    "connectionPacketsReceived": 99821,
    "connectionPacketsLost": 11039,
    "connectionBytesReceived": "151.52 MBytes",
    "connectionBytesSent": "0.64 MBytes",
    "connectionVideoLatency": 125,
    "connectionCodec": "VP9",
    "connectionVideoDecoder": "libvpx",
    "connectionResolution": "1280x720",
    "connectionFrameRate": 45
}

2. Audio & Video in different webrtcbins

{
    "connectionPacketsReceived": 85268,
    "connectionPacketsLost": 8331,
    "connectionBytesReceived": "125.72 MBytes",
    "connectionBytesSent": "0.43 MBytes",
    "connectionVideoLatency": 46,
    "connectionCodec": "VP9",
    "connectionVideoDecoder": "libvpx",
    "connectionResolution": "1280x720",
    "connectionFrameRate": 67
}

It doesn't seem like do-timestamp is doing much for me, the latency for the first two kept rising from 30 all the way to 100+ but in the third it stayed pretty much the same after a few mins. The problem is definitely in webrtcbin as everything else is the same. P.S. don't pay too much attention to the framerate. My network is very lossy so it varies a lot around 60.

Hi @Xosrov, do you have an implementation of using two webrtcbins in selkies-gstreamer? Could you publish this for others to try it?

But as I re-read your message I see that you said "I borrowed some of the code from your project and used it in ours". Which project are you referring to? Is there any code we could take a look at?

[Xosrov]

@alxlive Hey! I was referring to the javascript code for generating the reports in my post. The code adding the actual WebRTCbin element is in this C file. I don't have an implementation in selkies ready, but I can get a simple but working version done by the end of the week so we can compare.

[alxlive]

That'd be great! Would be awesome to confirm that your fix works in selkies too.

[ehfd]

That'd be great! Would be awesome to confirm that your fix works in selkies too.

Merged in main.

[ehfd]

For future reference, make sure to have the web interface ensure that the audio WebRTC stream and the video WebRTC stream are both initialized and running before starting the interface. Make sure to fail or reload if either one of the streams fails.

Sometimes, the audio stream fails solely and the video stream keeps going on without the audio.

This still hasn't been 100% fixed and is tracked at https://github.com/selkies-project/selkies-gstreamer/issues/109. However, it's just a matter of the web interface checking both audio and video streams are alive.

[ehfd]

https://groups.google.com/g/discuss-webrtc/c/ZvAHvkHsb0E https://groups.google.com/g/discuss-webrtc/c/yExd2Kou54c https://webrtc.googlesource.com/src/+/refs/heads/main/docs/native-code/rtp-hdrext/abs-capture-time/