Frame buffer option: Fixed delay

[YorVeX]

When synchronizing events or feeds (e.g. a separate audio feed) outside of OBS this can only be done when the delay between the sender and receiver is known and stays stable throughout a session.

With existing solutions like NDI or Teleport this is not possible, often the delay would start off at 50 ms but reach 200 ms 2 hours later.

This usually happens

1. slowly and constantly because of clock drift between the systems (so that frame timestamps differ)
2. as a sudden jump after a short lag in sender or receiver OBS occurred

That this happens cannot be avoided, but a frame buffer could be used to compensate for this. On a 30 FPS feed a video frame is shown for 33.33 ms. If the frame buffer is set to 500 ms, it would hold 15 frames. With the current implementation the frame buffer will always keep exactly those 15 frames.

With the new "Fixed delay" option if the delay of the feed increases by 33.33 ms or more the frame buffer would reconfigure itself to now hold only 14 frames, effectively handing a frame out to OBS for rendering 33.33 ms earlier, therefore countering the added delay and restoring the original resulting delay.

The same in the other direction, the buffer would be increased to hold 16 frames then.

All of this is an experimental idea, it needs to be tested how the switch can be done so that the frame jump caused by this has a minimal negative effect.

When decreasing the buffer an additional frame is sent to the OBS output, however, by the timestamp of that frame OBS would know it's not due yet and just queue it internally, since it is coming from an async source, and it wouldn't make any visual difference. So in addition the offset of the timestamps of all frames need to be adjusted too. This needs also be applied to audio frames, otherwise it would cause A/V desync.

On the contrary, when increasing the buffer one frame will simply not be sent to the OBS output when it would be due. This is what is already happening when a frame is skipped because of lag, the frame buffer should already have mechanisms to deal with that.

12 should be finished already as a base for this.

[Tomcatt]

I'm addressing audio syncing issues across various sources. Currently, the only workaround is to segregate the audio sources on the gaming rig. Typically, my preference is to centralize all the work on the computer that will stream the data to the global audience.

[YorVeX]

I'm addressing audio syncing issues across various sources. Currently, the only workaround is to segregate the audio sources on the gaming rig. Typically, my preference is to centralize all the work on the computer that will stream the data to the global audience.

I guess this feature could be interesting to you then. Unfortunately currently I can neither give a guarantee that it will work as intended nor give you an ETA, just stay subscribed here and you will find out when it's finished. Since I expect that there are not many people who would need such a feature I am happy about every tester I can get.

[Tomcatt]

So far, I'm impressed with what you've got here. I had to play with your compression ratio (which was really missing from NDI) but I'm able to stream 4k over my 1G network. I have a 10G adaptor coming so I can't wait to push this a bit more and record the results.

[YorVeX]

As long as you stick with the OBS default NV12 color format with 10G you should be able to transmit 4K video at 60 FPS completely without any compression (audio is transmitted raw anyway), meaning you save a lot of CPU resources on the sender (gaming) PC and retain the original quality 1:1.

[YorVeX]

While Beam doesn't provide an audio only filter yet, there is two things you can try:

1) If you have multiple separate audio and video sources that belong together (e.g. game capture + game sound and cam + mic) keep them together by running an OBS instance for each of them, you can use OBS portable mode to be able to run separate OBS instances simultaneously. If sent as one combined feed it's less likely to go out of sync (however, the combined game feed can still become out of sync with the combined cam+mic feed). Running an extra OBS instance will use extra resources on the gaming machine though and a lot more bandwidth for the video feed, so I don't know whether it's feasible for you. I have done this for a long time and got good results with it, but I only used 1080p60, if you go with 4K that's going to be a lot more demanding.

2) The hacky and funny solution assuming you use Teleport and the separate audio filter: after starting each OBS instance (sender and receiver) deliberately produce a short lag in OBS by quickly showing and hiding an image source with "Unload image when not showing" option set (1080p to 4K image should suffice). This way you will provoke a bigger portion of the delay that OBS usually builds up over time right from the start. Now sync audio on the receiving machine with that already-lagged feed using the usual ways to do this (audio sync offset or video delay filters, depending on the scenario). The delay might still change a bit from this point, but not so much as it often does when comparing freshly started OBS sessions to the same sessions 2 hours later. You might be lucky that it stays within a range that from human perception still looks like good A/V sync. Of course from this point on you always need to remember to produce this lag after starting each OBS instance, and there is the risk that also this initial delay that the lag causes differs from attempt to attempt. As I said, hacky. But most probably better than using a fresh session and waiting for it to change its delay over time, causing A/V desync.

[YorVeX]

As I was afraid this is becoming a real brainfuck.

My original idea of lowering the delay by just dropping a frame from the frame buffer and immediately handing OBS out the next frame doesn't work, because OBS then applies its own buffering logic. Instead of directly using the next frame I give it (which would effectively shorten the delay) OBS will just buffer this frame, freeze and wait one frame and then give out the next frame delayed so that it's back to its original timing and the delay stays the same - and from this point on keep on working with a buffer. OBS is buffering up to 30 frames this way, so only when I exceed those 30 frames I get a reset and an actual change of the effective delay.

The only way I can still think of to solve this is from this point on to fake all timestamps that I give to OBS. I skip frame 123 and directly hand out frame 124 to OBS, but I lie and tell OBS this would be frame 123. Then OBS will not buffer it and immediately use it. Which sounds simple at this first step becomes complicated really fast, because from this point on I will always have to tell OBS the current frame timestamp would be the one that in reality is the next. If I skip more than one frame, then this gap becomes even bigger. The problem doesn't stop there, because to keep it in sync I also have to do that for the audio timestamps, but they have an entirely different frame timing, so I can't just skip the same amount of audio frame timestamps like for the video timestamps.

My brain is just melting right now. Before I go down that rabbit hole I will try whether I can't just cause some kind of reset with OBS by sending it an intermittent 0 timestamp or so and hope that the visual and audio glitch this causes won't be too big.

[YorVeX]

Good gracious, I think I've done it. The timestamp manipulation actually seems to work, the delay can now be adjusted in both directions to make sure the feed always matches the configured delay.

[YorVeX]

Not closing this yet, as the current implementation only adjusts the buffer by 1 frame per second. This is to keep the glitches (which cannot be avoided when adjusting the buffer) low and is fine for cases where small delay differences need to be countered, e.g. by a small lag or a slow clock drift.

However, if there is a big lag that causes a big delay jump (in my tests I could get differences of 500 ms) these adjustments take very long: to counterbalance 500 ms the current logic needs 30 seconds of small glitches every second at 60 FPS. In such cases it would be better to just do one big jump, in this situation there is a lag anyway. This needs some testing though, as it's not yet clear to me how OBS reacts to it.

That said, even the current implementation will already successfully keep any delay stable, albeit with an unnecessarily long adjustment phase, so there's me being very happy and hoping I didn't miss something in my tests, but so far it looks really promising.

[YorVeX]

Bigger delay corrections don't work, the correction logic ends up needing to use even more correction steps in total than it does when going in small steps, probably due to how OBS reacts to it. I am still happy with the outcome, Beam is now the only OBS to OBS transmission plugin that can do this with a fixed, guaranteed delay (except for some short correction times).

[YorVeX]

At least this is working, so in the above worst case scenario of correcting 500 ms it will now take 15 seconds instead of 30.