Lack of JitterBuffering on input causes playback issues

[Roger-Roger-debug]

I've been trying to stream with webrtc for some days now but keep running into the same issue.

First up: I'm on Gentoo. Tried the Flatpak build from the linked site in #20 but that didn't work at all. After that I compiled obs from master, using the submodules on libdatachannel as opposed to native libraries. With this setup I could stream to broadcast-box on localhost and watch it just fine.

Opening port 8080 others could watch my stream as well, and streaming to my public ipv4 address worked without any issues either. However, I want to run broadcast box on my remote server and this is where the issue arises. I copied the files over and executed the same command to start the server but can't get any video to show up. To make sure it's not some issues with blocked ports I made an ssh tunnel forwarding port 8080 to my local machine and have broadcast box running there. Under this setup the stream works again.

I don't understand why it only works when the server is running my local machine. Both machines used the same command to start it (INCLUDE_PUBLIC_IP_IN_NAT_1_TO_1_IP=yes UDP_MUX_PORT=8080 APP_ENV=production go run .), using the same files and have the same go compiler running it.

My friend used the windows build from the linked issue and streamed to the same addresses I did, with the same results. When I could stream he could too, and when it was broken for me it was broken for him. The only difference was when we tried using https://b.siobud.com/, where his stream worked and my stream only had audio.

[Roger-Roger-debug]

With the power of print statements I found out that the ICEConnectionState goes from checking to failed after 20 seconds. However neither my go nor my webrtc knowledge are good enough to do anything with that information.

[Roger-Roger-debug]

Updated to the newest commit. The stream doesn't crash anymore on obs' side of things and I get audio, however video is still a black screen.

[Roger-Roger-debug]

I left it running for a bit longer. Audio works fine and I got a single frame transmitted after five minutes. After another five minutes I got around half a second of blurry frames and then the image stuck again. The only thing printed in the log is

SetRemoteDescription called with no ice-ufrag

[Sean-Der]

Sorry @Roger-Roger-debug I haven't responded sooner

Are you ingesting video from OBS? Can you lower the bitrate and see if it helps? What are your encoder settings?

[Roger-Roger-debug]

No worries. Yes, I'm ingesting from OBS (latest master) using ffmpeg vaapi h264 as the encoder with a constant bitrate of 2500 Kbps. I did try different settings without success.

[Sean-Der]

Can you try x264? I am not sure what the problem is, but hoping we can remove some variables.

Keyframe Internal - 1 second CPU Usage - veryfast Profile - Baseline Tune - Zerolatency

[Roger-Roger-debug]

It works much better with those settings. Now I get around 1fps with frames that look like this instead of nothing

I tried playing around with the bitrate a bit but it doesn't seem to make a difference.

[Sean-Der]

Next I would look in chrome://webrtc-internals at your inbound-rtp stats. These will be useful to start figuring out what part is failing.

[Roger-Roger-debug]

(I suspect the high packetsLots is causing the issues)

[Sean-Der]

I don't have a good answer on the why though sorry :/

A server along the way or the link itself is congested OR it really has deterministic packet loss?

I would try to make everything as small as possible (bitrate/resolution) until it works. Then work backwards from that. This is a weird one for sure :)

[Roger-Roger-debug]

I tried out some more settings (resolution, bitrate) but never got a single frame without heavy artifacts. When I find the time I'll bring the server into my LAN and try out some more stuff to see if it's somehow related to that.

[gavtroy]

I have the same behavior – it works locally but remotely (whether self-hosted or https://b.siobud.com/) there's a brief burst of video every few seconds. In my case inbound-rtp stats don't show any packetsLost and the bytesReceived_in_bits/s and keyFramesDecoded/s are consistent with expectation, yet framesReceived/s is all over the place. Graphs: remote, local

[gavtroy]

I can publish a webcam stream from Broadcast Box in one tab and watch it back in another tab without issue, so I don't think it's a playback issue. But I also don't think it's solely a problem with OBS because livestream-demo.livekit.io works flawlessly (albeit without audio).

Edit: I'm on Linux too, but now I got a chance to test on Windows with the same version of OBS (30 rc2) with the same settings and on the same network (and weaker hardware) and everything worked fairly well.

[gavtroy]

It looks like the main difference with Linux is that I have massively more out of order packets. Broadcast Box handles this very poorly compared to LiveKit. If I comment out https://github.com/Glimesh/broadcast-box/blob/8b218c02f051d819aa906e0298f3321d304f80b4/internal/webrtc/whep.go#L168 then the performance is drastically improved such that Broadcast Box and LiveKit perform about the same. Both do still have small issues regardless of Linux or Windows, but it looks like ~1% of packets are dropping and I suspect there's a bug on the OBS side causing this.

Logically I don't see an issue with timestamps being out of order, but, based on these findings, in the real world clients must rely on timestamps being monotonic with reference to sequence numbers. In any case, shouldn't it be fine to just pass the packets through with the original sequence number?

[RedMser]

@gavtroy Thanks for sharing your workaround! In our experience, it also fixed the visual artifacts similar to those found in the picture further up. However in our case, they happened every minute or so only, not continuously.

Even with the code change though, the sporadically appearing visual artifacts are replaced with the stream freezing for around 8 seconds before it catches up... I don't have a proper way to debug the exact problem, which is why I did not open an issue for this yet. Streaming with OBS 30 but also happens with /publish. Only for one user, so it's likely a network configuration/firewall issue.

Just wondering if you have similar problems or if your workaround was enough to completely fix all issues for you?

[Sean-Der]

Broadcast Box is re-stamping the packets for Simulcast. If the viewer switches between layers the Sequence Numbers need to stay in-order.

We need a basic jitterbuffer in Broadcast Box. @gavtroy can you easily reproduce?

If you can I would love to add one to Pion (and have Broadcast Box use it). Then have you test :)

[gavtroy]

Yes it's still an issue for me. I'd be happy to test it out when you have something to try (and after Christmas).

[jhvst]

I had similar problem, where the stream would freeze every few seconds. Using the OBS options above but changing buffer size to 500 seemingly fixed all issues. I also added -framedrop to the x264 options, but I'm unsure whether this actually has any effect.

OBS version: OBS Studio - 30.1.0. Both client and server are Linux. Server uses Docker.

[Sean-Der]

This was addressed by aeb4043d3e2e5a5973ebf77bc88e426ffbde6f33

@Roger-Roger-debug @jhvst Would you mind testing again?

@RedMser do you still see it happening?

@gavtroy catch you on discord! It sounds like this doesn't fix your issue though, I think we will have to debug 1:1. I am not sure what I need to improve next

[Roger-Roger-debug]

Did a quick test with flatpak obs and an ssh tunnel to my server which didn't work (but I think didn't work in my original setup to begin with). I can't test more than that for the foreseeable future I'm afraid.

[RedMser]

@RedMser do you still see it happening?

Yes, the stream still does seem to randomly (every minute or two) freeze for 10-20 seconds, not sure what causes it. I don't have much time or knowledge to debug this sadly, but I'll share more details in a new issue if we ever find out a root cause.

[Sean-Der]

@RedMser are you using OBS to publish or browser/other tool?

[RedMser]

Using OBS was when we had issues. Did not try browser publish page, but streaming e.g. RTMP or SRT with OBS works fine.

Next time we try I'll check the webrtc stats page, OBS stats graphs etc. and see if we notice anything...

[Sean-Der]

I believe you are hitting https://github.com/obsproject/obs-studio/issues/10564

the next release of OBS will handle it properly. Unfortunately nothing Broadcast Box can do to help :/