Audio Video sync is lost over time

[adrian-j-programmer]

When stream is running for longer periods of time in a bad performance environment, the audio video synchronization is being lost gradually. This can also be observed to happen during performance hitches that happen when many assets stream in at the same time. A good example of this behavior is teleporting to a completely different part of the level where a lot of assets (including the millicast stream). Even though the stream could be in perfect sync before we teleport, after teleporting the AV sync would be lost.

[dbaldassi]

I will try to reproduce this on the test project you provided me.

[dbaldassi]

So far, I've not been able to reproduce it. I've noticed that the audio timestamps are not set by libwebrtc and I'm trying to figure out why. I will check this with Unreal 5.1 since they have updated libwebrtc to a more recent version (M96)

[dbaldassi]

I'm starting to think that the AudioComponent can be stop by unreal if it prioritize something else in a context of bad performance. The AudioBuffer would still enqueue audio data while it is not being played, hence the desync. I will add a check to prevent this. Do you know if audio was behind video when you experienced this ?

[adrian-j-programmer]

I'm pretty sure it would gradually start lagging behind the video track, but I actually can't test this again today to confirm.

[adrian-j-programmer]

I just confirmed, audio is behind the video when they get out of sync.

[maximeLong]

Hi @dbaldassi, could you post your team's current progress on this issue?

[dbaldassi]

From my last comment, I have added the check to not queue audio data if the UAudioComponent is stopped, but Adrian still seems to have reproduced the issue after that. We are able to reproduce this issue well on a computer with low performance, then losing the focus of the editor (going to a second monitor for example). After ~10mns like this we observe the de-sync. Otherwise, we are still investigating on this issue.

[adrian-j-programmer]

Audio is still out of sync at the start of play in more complex levels if the stream is playing right from the get go. It also still get out of sync gradually over time, but it takes significant amount of time to reproduce.

[maximeLong]

Hi @dbaldassi has there been any progress on this over the past week?

[dbaldassi]

I'm currently writing some debug logs to log info about the plugin state when the issue is occuring as Adrian suggested, since we struggle to reproduce the issue on our side. Also, I'm working on gathering the webrtc stats to inspect the states of the webrtc audio and video jitter buffer, and log those stats into a file that we can later load and analyse so we can see if there is something wrong with the libwebrtc implementation shipped in unreal.

[maximeLong]

Makes sense. But I'm confused about your comment above:

We are able to reproduce this issue well on a computer with low performance, then losing the focus of the editor (going to a second monitor for example). After ~10mns like this we observe the de-sync.

Is reproducing in this condition not giving your team enough data to track down the issue?

[dbaldassi]

I will copy/paste the message of my teamate who did reproduce, but it was just only once. It might be because he was publishing with obs on the same low performances computer

In the meantime UE has processed the project enough for me to get consistently high FPS until I set the quality to Cinematic. I left the stream on for over 30 minutes looking directly at the stream at >50 fps and it didn’t get desynced at all. I then checked for about 20 minutes while looking away at some buildings and sometimes randomly moving the mouse which caused fps to drop to <20 and still the stream was always in sync. I then alt-tabbed out of UE, waited maybe 3 seconds and when I refocused audio was already slightly late.

I can also observed that, after over-complicated the scene, and leaving the focus to another window, then coming back after ~10mns, I can see that the audio is slightly late, but it comes back in sync after a minute or so. That is why I am trying to log the state of the rtc buffers, to see if they are responsible for this behavior.

[dbaldassi]

I have added logs and WebRTC Stats to the plugin. (using the csv profiler) 258ea42 Can you record a session of the issue and send me the output log file and csv file ?

[maximeLong]

Yes, we can do this. Thanks @dbaldassi

[adrian-j-programmer]

@dbaldassi Hey, could you make a version of this commit that's compatible with 5.0.3? Unfortunately it looks like this specific branch only compiles with 5.1. Thanks!

[dbaldassi]

ok, I will take care of this.

[dbaldassi]

I have backported the commit here. https://github.com/millicast/millicast-player-unreal-engine-plugin/tree/debug-submodule-UE5