Video players have an audio de-sync.

[RaliiTheAvali]

Describe the bug

Video players appear to get de-synced from the audio, audio appears to progress ahead of the video itself in almost all cases. Long videos (i.e. 15+ min. or even movie length). Audio de-sync seems to happen at random intervals across different users. Unknown if Internet speeds affect this, or if PC hardware/software is a contributor. Occurs when there is multiple people in a world/session. (!!More info will be added as i gather PC and Net info from users!!)

Relevant issues

N/A

To Reproduce

Steps to reproduce the behavior:

1. Get a group of people or friends together and any random video in 15+ min length ( YouTube, personal download, etc...) preferably one where mouth movements for speech can be seen, as it's easier to spot a de-sync.
2. Watch video and keep an eye out for a de-sync in audio with the video ( almost always audio starts to get ahead of the video around the 2-4+ min mark.)

Expected behavior

For videos to remain in sync with themselves with out the need to stop start a video every few min.

Screenshots / Video

N/A

Bug information (please complete the following information):

• How often does it happen: Always
• Does the bug persist after restarting Neos? Yes
• Neos Version: 2021.7.2.1345
• Neos Platform: Any
• Regression: Unknown

Additional context

A temp but annoying fix is to pause and start the video, can be come cumbersome when trying to watch a full length movie with friends or other users ( I am attempting to host a Cinema on Neos a few times out of the month)

PC specs of users who have noticed the de-sync Issue Neos Username: Ralii Discord Username: Ralii#2181

Video/audio De-sync: 4-5+ min mark

PC spec.s- CPU: Ryzen 5 3600 GPU: EVGA | Geforce RTX 2070 Ram: 32GB DDR4 3600 MOBO: MSI MPG X570 GAMING PLUS How is my pc getting net?: Ethernet Cable OS: Windows10

Internet- ISP: Xfinity/Comcast Download speed: 200 MBPS Upload speed: 10 MBPS

Reporters:

Ralii / Ralii#2181, Gawdl3y / Gawdl3y#4269, (more to be added as User info becomes available)

[Frooxius]

Which playback engine does this happen with? Is it the Unity Native or just the libVLC?

The current implementation of libVLC that we're using doesn't support proper audio sync and as such it has drift with some sources. We're planning to switch to an official implementation which has proper audio synchronization mechanism that shouldn't exhibit this issue. As such we won't be fixing it for the old one.

Unity Native playback shouldn't exhibit the issue either.

[RaliiTheAvali]

Which playback engine does this happen with? Is it the Unity Native or just the libVLC?

The current implementation of libVLC that we're using doesn't support proper audio sync and as such it has drift with some sources. We're planning to switch to an official implementation which has proper audio synchronization mechanism that shouldn't exhibit this issue. As such we won't be fixing it for the old one.

Unity Native playback shouldn't exhibit the issue either.

Let me get some testing in, and check the player currently in-use, and to stress test both options. an update will be provided in about a week. i'll close the ticket if the Native playback doesn't have an issue, seeing as the libVLC player has a change in the future.

[Gawdl3y]

I can confirm that the same issue happens with the Unity native player as well. I've been running movie/show-watching nights around weekly/twice per week in Neos for about two months now, and nearly everybody participating experiences this issue. The audio gradually becomes out of sync with the video, drifting over the course of several minutes. Usually, we find ourselves quickly pausing then playing the video every 3-4 minutes on average. Sometimes it drifts more quickly, and sometimes for some people it takes a very long time before the desync is noticeable. We usually are using libVLC, but have confirmed the same desync behaviour with Unity native on more than one occasion.

Thanks for creating this ticket, Ralii, I've been meaning to for a while, but kept forgetting/getting distracted.

Neos username: Gawdl3y Discord tag: Gawdl3y#4269

Video/audio de-sync: Varies, usually between 3-4 minutes before it loses sync by about half a second to a second

PC specs

CPU: AMD Ryzen 9 5950x (and Intel i7-7700k before last Friday) GPU: AMD Radeon 6900XT (and Nvidia GeForce GTX 1070 before a few weeks ago) RAM: 16GB DDR4-3200 Motherboard: MSI MEG X570 Unify (and an Asus board before last Friday, forget which, probably not relevant though) How is my pc getting net?: Ethernet cable OS: Windows 10 64-bit, build 20H2

Internet

ISP: Cincinnati Bell FiOptics Download speed: 1000Mbps Upload speed: 250Mbps

I also cleaned up that info template a bit, here's the source if anybody else wants to use it:

**Neos username:** 
**Discord tag:** 

**Video/audio de-sync:** 

### PC specs
**CPU:** 
**GPU:** 
**RAM:** 
**Motherboard:** 
**How is my pc getting net?:** 
**OS:** 

### Internet
**ISP:** 
**Download speed:** 
**Upload speed:** 

[Gawdl3y]

Actually, now that I think about it, it's possible the attempts to switch to Unity native were unsuccessful and we were still using libVLC when we thought otherwise. I was using the "Force Unity Native" button to switch to it, and the video player went blank for a second like it switched, but I never actually verified this switched it away from libVLC.

[Hayden-Fluff]

I personally experience issues with both playback engines with Unity being the less impactful of the two, VLC consistently gives me CPU issues since I am majorly CPU bound and on a laptop with a poor thermal design, so encountering thermal throtteling with a VLC video loaded isn't uncommon, while Unity just freezes the video while frames are dropped (I assume this is because Unity playback relies on the main thread) I haven't noticed any desync issues with Unity, (if I have I can't recall it) which is another reason it's preferred in most situations, but in almost every case I've observed of VLC being used I have observed some kind of desync after anywhere from 2-5 minutes of playback.

Neos username: Hayden Discord tag: Hayden#3180

Video/audio de-sync: As mentioned above, anywhere from 2-5 minutes of playback and gets worse over time if not corrected, can be anywhere from barely noticeable to over a second of delay.

PC specs

CPU: AMD Ryzen 7 3750h GPU: Nvidia GeForce RTX 2060 mobile RAM: 16GB LPDDR4 Motherboard: laptop How is my pc getting net?: Ethernet OS: Windows 10 20H2

Internet

ISP: TDS Download speed: 100 MB/S Upload speed: 10MB/S

[Frooxius]

@Hayden-da-Serg I'm a bit confused by your statement. You say that you experience issues with both playback engines, but then that you haven't noticed any desync issues with Unity. Are you referring to other issues with Unity one?

If there are desync issues with Unity one then it's something deeper, probably at system level, because of how the audio synchronization is handled by it (it uses push, rather than polling).

@Gawdl3y Yeah the Unity Native won't get used if Neos isn't certain that it can handle the media, even if you force it. Are the desync issues only ever occurring with libVLC active then?

If so, this should get fixed after we switch to the official implementation.

[Gawdl3y]

I've remuxed one of the video files from MKV to MP4 (no encoding difference, just container format change) for testing the Unity native player, but Neos/Unity really doesn't like the audio when playing via Unity native. It's encoded as H.264 + AAC as Unity recommends, but playing the video causes awful popping/hissing to pretty much take over the audio buffer. Playing with libVLC (or external, non-Neos players) doesn't exhibit that particular issue, but that's not really a solution when the whole point is to test Unity native. I'm suspecting that Neos/Unity isn't behaving well with the audio due to one or more of sample rate (48KHz), bitrate (640Kbps), or number of audio channels (6), so will need to transcode it a bunch to narrow that down. Don't suppose you know the ideal settings for this off the top of your head, do you?

[Frooxius]

The Unity Native is very sensitive to particular encoder settings too. It won't play any H.264 stream correctly, you pretty much need to match the right encoder profile, bitrate and other settings for it to work well. I'm not sure what exactly that is, but generally whatever YouTube videos use is good.

You don't need to test the video specifically though, all I need to know is if you ever encountered the audio desync with Unity Native active or if it was always libVLC. If it was always libVLC, then it makes the problem much simpler, but if it was Unity Native as well then there's some deeper problem somewhere.

[RaliiTheAvali]

The Unity Native is very sensitive to particular encoder settings too. It won't play any H.264 stream correctly, you pretty much need to match the right encoder profile, bitrate and other settings for it to work well. I'm not sure what exactly that is, but generally whatever YouTube videos use is good.

You don't need to test the video specifically though, all I need to know is if you ever encountered the audio desync with Unity Native active or if it was always libVLC. If it was always libVLC, then it makes the problem much simpler, but if it was Unity Native as well then there's some deeper problem somewhere.

On another side note, Unity player doesn't support Live video feeds. Back to topic, We'll have an answer for this sometime after Sunday. a Movie night is going to be hosted in a headless session 'The Scratch' and we'll attempt to use the Unity player for the viewing, unless of course the audio issue Gawd3ly mentioned becomes an issue.

[Frooxius]

On another side note, Unity player doesn't support Live video feeds.

That's expected behavior. Unless the stream codec format can be detected in advance (e.g. through metadata from youtube-dl), Neos will always default to libVLC.

[RaliiTheAvali]

On another side note, Unity player doesn't support Live video feeds.

That's expected behavior. Unless the stream codec format can be detected in advance (e.g. through metadata from youtube-dl), Neos will always default to libVLC.

An unplanned movie night prompted us to go ahead and play using Unity player, no one experienced an audio de-sync. Since you have mentioned that you have plans to update the LibVLC player in Neos we can close this bug down. Out of curiosity is there an expected/estimated time frame of when the player will be swapped out?

[Frooxius]

I plan on trying the swap soon, all of the issues with the new system should be unblocked now, unless I run into more new problems

[shiftyscales]

The new libVLC library is still a fairly high priority, it's currently awaiting implementation of the rendering features Geenz is working on now. I imagine shortly after those are released, finishing the swap should be possible.

[AshtonSparx]

Was this on the headless server that was causing the desync @RaliiTheAvali

[RaliiTheAvali]

Was this on the headless server that was causing the desync @RaliiTheAvali

It was both.

[Frooxius]

The desync between audio and video is local only, due to the UMP libVLC playback engine decoding the video when playing for given user, so it wouldn't be affected by headless. It'll be solved with the upgrade to the official libVLC library.

[Enverex]

As a workaround for anyone else that finds this bug while waiting for a proper fix;

The least intrusive way to resync your audio/video (without interrupting anyone else in the process) is to switch worlds back to local home, wait a few seconds then switch back to the world with the video. This corrects the sync 100% of the time for me (until it eventually desyncs again for whatever reason).