Short lag on the sender side when sending using a filter causes constant A/V desync

[YorVeX]

Scenario to reproduce:

• Start with a vanilla OBS 27.2.4 instance and install the latest Teleport plugin to it.
• Run the OBS instance and do some base configuration fitting your setup, e.g. disable mic and set audio devices correctly.
• Add an Image Source with "Unload image when not showing" enabled and add a large image to it, e.g. one from here.
• Add a Media Source with this video loop and the "Loop" setting enabled. Mute this source and just to be sure move it out of the canvas so that it's playing but not visible.
• Add a Teleport filter to the Media Source with the A/V test video loop and enable this.
• Add a Teleport source tuned into the Teleport feed that was just configured.
• The video loop should now be playing through the Teleport source perfectly in sync.
• Now click the eye icon next to the Image Source multiple times in rapid succession to have the constant loading and unloading cause lag, watch the OBS Stats window to observe this if you want to be sure. Then stop and wait until no more frames are dropped.
• Result: The video loop displayed through the Teleport source now has A/V desync
• Now hide the Teleport source and then show it again, this will make the feed become in sync again

Some additional notes:

• The same occurs if a similar scenario is repeated with separate sender and receiver instances on different computers.
• The same applies to audio only Teleport streams.
• What causes the lag doesn't matter, in another scenario it was triggered by a Game Capture source that is lagged for a second when tabbing into and out of a game.
• Doing a similar test with two separate OBS instances and using the base stream through Tools -> Teleport (instead of a filter) to send out the stream does not show this behavior. After the lag ends the A/V sync completely recovers every time.

[YorVeX]

Using a render delay filter is pretty much inevitable when working with separate video and audio streams that need to be synced and lags can occur even in the best setups (pretty much everyone knows the lagging Game Capture story) so this bug really hurts.

There is a workaround (hiding and showing the Teleport source) but that can only always be manually applied when one notices that this occurred in the first place. OBS log also doesn't show anything, also none of the infamous "audio buffer increased" messages (which is why I believe that this is not related to #15 ).

What would already help a lot is when the plugin could detect itself that it was lagged (just like that video delay filter) and then reset itself the same way it is reset when quickly hiding and showing it again, so that also its filter chain is reset. I mean, at that point everything is lagging and flickering anyway.

A lot better of course would be if lags on the sender side wouldn't affect the receiver side in that way at all. Maybe if the receiver doesn't get new frames from the sender on time because it's lagged just repeat the last frames in the meantime? And if no frames are received for an even longer period of time don't show anything anymore? Would something like that be possible? Anything that makes sure the rendering flow on its host OBS or filters applied to it is not disturbed in anyway should solve this issue.

[fzwoch]

You may be dealing with two things here. I haven't read trough it completely, but I know there is something happening with a media source and looping alone. E.g. have that video playing in a loop, put a teleport filter on it, receive it and have them displayed side by side. They start out okay-ish but then at some point (probably after looping) the teleport stalls for a bit and continues, but has increased latency compared to the source. I suspect the media loop does something weird.. but I cannot say what.

[YorVeX]

I have just repeated the test without the delay filter and can confirm that loading/unloading the image source also messes up the sync timings (and hiding/showing the Teleport source resets it). Maybe then it was a coincidence that the delay filter was countered almost perfectly for me and it doesn't actually have anything to do with it.

The same happens for a Teleport stream with cam and mic audio so I don't believe that this happens only for looped videos in Media Sources. What triggers the lag probably isn't important, only that it is triggered. About the video loop issue I don't know, maybe when the loop in a Media Source repeats this also triggers a short lag?

[YorVeX]

Since my initial assumption was wrong that this is related to the delay filters I edited the title and description to reflect my latest test results. This issue is now about the fact that these lags break the filter based Teleport streams (while the same thing does not break a Teleport base stream through Tools -> Teleport).

Since lags can always occur especially during longer sessions this renders filter based Teleport streams pretty much useless right now, they will eventually all go out of sync and would need constant monitoring and manual resyncing by resetting the sources to keep in sync.

[fzwoch]

Well, maybe you have found the reason why NDI gets out of sync. All we can do is pass through data with corresponding timestamps. If OBS trips over trying to render it correctly there is not much we can do.. unless someone can tell what a plugin is do it in such cases. I wouldn't know.

[YorVeX]

At least in my setup with NDI a lag kills also a stream sync that is not filter based (in 20 out of 20 cases where I produced a lag), while with with Teleport the sync of a non-filter based stream will heal itself a second after the lag stopped (again in 20 out of 20 attempts). So Teleport definitely does something right where NDI goes wrong. Based on that I would hesitate to conclude that the filter based solutions have the same problem. Also for NDI I could definitely reproduce that Render Delay filters become ineffective after a lag, which is the reason why I jumped too quickly to the conclusion that for Teleport it would be the same issue when it turned out to actually be a different problem here.

About the timestamps I must admit that I don't really understand how this is supposed to work. My starry-eyed view would be that with proper use of them you could send video and audio separately, having them travel different routes through a network and e.g. the video arrives 200 ms later than audio but using the timestamp it can be put together again correctly on the receiving and. With a system like this it should never become desync no matter what.

Maybe it's just me not knowing Go but in the code I only ever see timestamp fields getting assigned, but there is never any logic applied to it. So OBS is supposed to be doing something meaningful with it?

[YorVeX]

Thanks for working with me through this and your time and effort! What you produced with Version 0.5.0 is a small miracle! Despite the base being provided by OBS being already vulnerable to lag induced sync issues as I am showing here you have managed to mitigate many lag desync scenarios with this update. In my advanced test scenario where the A/V feed is sent from OBS 1 through OBS 2 to OBS 3 when I now produce lag on OBS 1 neither the recording nor the audio monitoring (especially the latter is remarkable) on OBS 3 lose A/V sync anymore. This also means that regarding stability/robustness Teleport is now even more ahead of OBS-NDI than it already was before.

The few remaining scenarios I believe can either not be fixed from Teleport or if so, it would be really hard to find out how. Either way, what I originally intended by opening this issue was fully addressed so I consider this closed, thanks again!