A/V desync after long period of playback with `--ao=wasapi`

[sunpenghao]

Important Information

Provide following Information:

• mpv version: mpv-x86_64-v3-20231008-git-78719c1
• Windows Version: Windows 11 22H2 with latest updates
• Source of the mpv binary: shinchiro builds
• If known which version of mpv introduced the problem: N/A
• Possible screenshot or video of visual glitches: N/A

Reproduction steps

With the settings in the log file below, there is noticeable A/V desync after about 12 hours of playback where video lags behind by about half a second. A seek clears this desync, but reaching EOF after continuous playback and looping over to the next file in the playlist does not help (I have 13 24-minute videos in my playlist). I'm able to reproduce the issue on a desktop connected to a 4K/120Hz TV and a laptop connected to a 4K/60Hz monitor. I have tested with --video-sync=audio and --video-sync=display-vdrop. Both have the same issue.

Log file

mpv.log

Sample files

Don't know if it helps but here is the MediaInfo of one of the files in my playlist:

General
ID                                       : 0 (0x0)
Complete name                            : 1-01.m2ts
Format                                   : BDAV
Format/Info                              : Blu-ray Video
File size                                : 7.19 GiB
Duration                                 : 23 min 42 s
Overall bit rate mode                    : Variable
Overall bit rate                         : 43.4 Mb/s
Maximum Overall bit rate                 : 48.0 Mb/s
Frame rate                               : 23.976 FPS

Video
ID                                       : 4113 (0x1011)
Menu ID                                  : 1 (0x1)
Format                                   : AVC
Format/Info                              : Advanced Video Codec
Format profile                           : High@L4.1
Format settings                          : CABAC / 4 Ref Frames
Format settings, CABAC                   : Yes
Format settings, Reference frames        : 4 frames
Codec ID                                 : 27
Duration                                 : 23 min 42 s
Bit rate mode                            : Variable
Bit rate                                 : 37.1 Mb/s
Maximum bit rate                         : 40.0 Mb/s
Width                                    : 1 920 pixels
Height                                   : 1 080 pixels
Display aspect ratio                     : 16:9
Frame rate                               : 23.976 (24000/1001) FPS
Color space                              : YUV
Chroma subsampling                       : 4:2:0
Bit depth                                : 8 bits
Scan type                                : Progressive
Bits/(Pixel*Frame)                       : 0.746
Time code of first frame                 : 00:59:59:0000000000
Stream size                              : 6.14 GiB (85%)

Audio #1
ID                                       : 4352 (0x1100)
Menu ID                                  : 1 (0x1)
Format                                   : PCM
Format settings                          : Big / Signed
Muxing mode                              : Blu-ray
Codec ID                                 : 128
Duration                                 : 23 min 42 s
Bit rate mode                            : Constant
Bit rate                                 : 2 304 kb/s
Channel(s)                               : 2 channels
Channel layout                           : L R
Sampling rate                            : 48.0 kHz
Bit depth                                : 24 bits
Stream size                              : 391 MiB (5%)

Audio #2
ID                                       : 4353 (0x1101)
Menu ID                                  : 1 (0x1)
Format                                   : PCM
Format settings                          : Big / Signed
Muxing mode                              : Blu-ray
Codec ID                                 : 128
Duration                                 : 23 min 42 s
Bit rate mode                            : Constant
Bit rate                                 : 2 304 kb/s
Channel(s)                               : 2 channels
Channel layout                           : L R
Sampling rate                            : 48.0 kHz
Bit depth                                : 24 bits
Stream size                              : 391 MiB (5%)

[Dudemanguy]

So I tried this out with roughly 12 hours of playback (big playlist with a lot of episodes) using --ao=pipewire on linux and --video-sync=audio. I didn't see any desync. It was just fine. I quit and loaded up the same episode again to be sure and it was the same. pipewire is a pull ao though so it makes sense that it would not be affected by this. Right now, I'm doing the same thing with alsa (a push ao like wasapi) to see if this happens or not. If it does, it's probably safe to assume this is push-ao specific but if not then it's probably something to do with wasapi. I'll report back in another ~12 hours.

[sunpenghao]

I'm doing the same thing with alsa (a push ao like wasapi) to see if this happens or not. If it does, it's probably safe to assume this is push-ao specific but if not then it's probably something to do with wasapi.

Cool, I'll start a test with a different --ao on my side as well.

Also, not really a fix to this issue but may I suggest that we somehow change the logic of file loading so that it resets A/V sync as seeking does? At least in this case desync doesn't build up indefinitely without manual seeks.

[Dudemanguy]

may I suggest that we somehow change the logic of file loading so that it resets A/V sync as seeking does?

That would be pretty hacky and would basically be invoking a seek whenever a new file is loaded (this would be really nasty with gapless audio). I assume the seek works since it resets everything. But someone could be watching a 12 hour video or something. It wouldn't really help them.

Edit: Also if you're really bored and want to double check, the offending commit is most likely this one: b74c09efbf7c6969fc053265f72cc0501b840ce1

[Dudemanguy]

Okay so after a little over 12 hours of playback using --video-sync=audio and --ao=alsa, I couldn't reproduce the issue either. So probably this is wasapi or windows-specific somehow. Either that or whatever error is just much bigger there and it would take longer to encounter noticeable desync on another platform.

[sunpenghao]

Just started another test with --ao=openal (seems to be the only working option other than wasapi) and I'll check again tomorrow. Hope this can narrow down the problem a bit.

[Dudemanguy]

Just to correct myself earlier, wasapi is a pull AO not push. Not sure why I got that backwards in my head.

[sunpenghao]

All right --ao=openal is working perfectly fine. Guess it's safe to say this is a wasapi issue.

[Dudemanguy]

One thing I did notice is that wasapi is the only AO that uses ao_read_data_converted instead of just plain ao_read_data.

[kasper93]

After 12h of playback the audio clock drift may be significant, but looking at the code it looks ok. I made small changes, to improve it a little, but in practice it wouldn't matter.

@sunpenghao Could you try with this diff applied? I didn't see "insane" correction in your log, but depending on hardware I think it actually may happen after long period, but probably a lot longer than 12h.

Also additional question, does the de-sync increase gradually? Or it starts after a while, which would mean we overflow somewhere?

EDIT: Also revert f2301d542e374a4651b0a961a24b1fe85f9cfefa because it will probably do funky things.

diff --git a/audio/out/ao_wasapi.c b/audio/out/ao_wasapi.c
index 66fc9b84cd..f7c72b5312 100644
--- a/audio/out/ao_wasapi.c
+++ b/audio/out/ao_wasapi.c
@@ -54,8 +54,9 @@ static HRESULT get_device_delay(struct wasapi_state *state, double *delay_ns)
     UINT64 sample_position = uint64_scale(position,
                                           state->format.Format.nSamplesPerSec,
                                           state->clock_frequency);
-    INT64 diff = sample_count - sample_position;
-    *delay_ns = diff * 1e9 / state->format.Format.nSamplesPerSec;
+    assert(sample_count >= sample_position);
+    UINT64 diff = sample_count - sample_position;
+    *delay_ns = diff / (double)state->format.Format.nSamplesPerSec * 1e7;

     // Correct for any delay in IAudioClock_GetPosition above.
     // This should normally be very small (<1 us), but just in case. . .
@@ -63,14 +64,9 @@ static HRESULT get_device_delay(struct wasapi_state *state, double *delay_ns)
     QueryPerformanceCounter(&qpc);
     INT64 qpc_diff = av_rescale(qpc.QuadPart, 10000000, state->qpc_frequency.QuadPart)
                      - qpc_position;
-    // ignore the above calculation if it yields more than 10 seconds (due to
-    // possible overflow inside IAudioClock_GetPosition)
-    if (qpc_diff < 10 * 10000000) {
-        *delay_ns -= qpc_diff * 100.0; // convert to ns
-    } else {
-        MP_VERBOSE(state, "Insane qpc delay correction of %g seconds. "
-                   "Ignoring it.\n", qpc_diff / 10000000.0);
-    }
+
+    *delay_ns -= qpc_diff;
+    *delay_ns *= 100.0;

     if (sample_count > 0 && *delay_ns <= 0)
         MP_WARN(state, "Under-run: Device delay: %g ns\n", *delay_ns);
@@ -116,7 +112,7 @@ static bool thread_feed(struct ao *ao)
     hr = get_device_delay(state, &delay_ns);
     EXIT_ON_ERROR(hr);
     // add the buffer delay
-    delay_ns += frame_count * 1e9 / state->format.Format.nSamplesPerSec;
+    delay_ns += frame_count / (double)state->format.Format.nSamplesPerSec * 1e9;

     BYTE *pData;
     hr = IAudioRenderClient_GetBuffer(state->pRenderClient,

[sunpenghao]

Also additional question, does the de-sync increase gradually? Or it starts after a while, which would mean we overflow somewhere?

It increases gradually. After about 3 or 4 hours the desync becomes noticeable, and after 12 hours intolerable.

I'll give a try to your patch, but it's probably going to be a few days before I can report back.

[kasper93]

It increases gradually. After about 3 or 4 hours the desync becomes noticeable, and after 12 hours intolerable.

Ok, then this patch would not change anything. I need to dig into it when I have a lot of time. Our audio clock is drifting apparently, but the question is how to properly handle that. At this moment I'm not sure QPC adjustment is correct. Also not even sure it is wasapi issue itself, but probably.

What happens is wasapi clock_frequency is constant during playback duration, while obviously real hardware clock will drift a little. And QPC is there to extrapolate this drift and adjust the timing, but I really looked into it half an hour and need more to diagnose what can we do.

[sunpenghao]

Sorry to have taken so long to follow up.

I tried kasper's diff and can confirm it doesn't fix the issue.

Also some new findings. On a playlist with only 60 FPS videos there is no desync even after ~18 hours. Later I'll try other frame rates and see if there is any pattern that might be of help.

[zhengqwe]

wasapi has two methods: Event and Push. Which one does mpv use?