Account for PTS delay in audio callback?

[jesse-trana]

Recently I ran into a usability issue around the audio callback. I've had the audio callbacks setup for quite some time now and things are working well, much in the spirit of the example here: https://github.com/mfkl/libvlcsharp-samples/blob/8b1d31e3238935d6fda406b0d7a398f47ec498d6/AudioCallbacks/Program.cs#L49

However, I introduced the flag "--ts-no-trust-pcr" and found that the audio and video became desynchronized by a large amount, nearly 2 seconds. Digging into it, I found that the audio callback was now getting called quite "early" whereas - by luck - it had been called close enough to "on time" previously that the effect was not noticeable.

I looked at how I should handle the situation to introduce a delay, and one of the key questions was: "How can I synchronize PTS back to the video timeline?" It appears that a key function, libvlc_delay could be used in conjunction with the pts argument; I was unable to find an alternate way to handle this as the input to the audio callback is in PTS and the playback timeline is I believe reported in wallclock time. Note that manually setting the audio delay on the MediaPlayer did not seem to resolve the issue.

Ultimately I ended up with something like the following hacky solution:

        [DllImport(/*Constants.LibraryName - inaccessible; relatively safe to hardcode on Windows*/ "libvlc",
            CallingConvention = CallingConvention.Cdecl,
            EntryPoint = "libvlc_clock")]
        private static extern Int64 LibVlcClock();

        private static Int64 EmulatedLibVlcDelay(Int64 pts)
        {
            return pts - LibVlcClock();
        }

        private void PlayAudio(IntPtr data, IntPtr samples, uint count, long pts)
        {
            //Terrible hack to delay until the right PTS (would be better handled further downstream or via other means)
            var requiredDelayMicroseconds = EmulatedLibVlcDelay(pts) - AUDIO_LATENCY_MS*1000;
            if (requiredDelayMicroseconds > NOTICEABLE_AUDIO_DELAY_US)
            {
                Debug.WriteLine("Audio/video sync gap. PTS: " + pts+ " Microseconds until play time of that PTS: " + requiredDelayMicroseconds); // We see 50+ ms gaps here
                Thread.Sleep(new TimeSpan(requiredDelayMicroseconds * 10)); //Going from microseconds to ticks (100-nanoseconds)
            }

            //Copy/play the audio
            //...
            Marshal.Copy(samples, m_AudioScratchBuffer, 0, byteCount);
            m_AudioPlayer.Play(m_AudioScratchBuffer, 0, byteCount); //The delay here is AUDIO_LATENCY_MS from above
        }

This is bad for a couple reasons: an extra function has to get exposed from libvlc, and the waiting methodology is terrible. Does libvlcsharp expose a better way to do this? I'd love to see an updated example that handles this nuance of the design.

Thanks for your time and consideration in this request.

[mfkl]

This is bad for a couple reasons: an extra function has to get exposed from libvlc

We can add the libvlc_clock to LVS. Nobody ever requested it so that's mainly why it isn't exposed.

the waiting methodology is terrible.

Yes Thread.Sleep is not ideal but does this code work the way you want it to?

I'm not sure there is a better way to do this. Task.Delay would do essentially the same thing.

Maybe there is a way to use the PTS data from libvlc in NAudio, using a delay function of theirs or something like outputDevice.DesiredLatency? Not sure.

OBS is using similar code. https://github.com/obsproject/obs-studio/blob/ba77ca65927b4a7666f48bc5d4ec054d36bacf93/plugins/vlc-video/vlc-video-source.c#L497

[jesse-trana]

Hey thanks for the response @mfkl !

If there isn't a much nicer way to do this, I almost wonder if it might not be better to expose or emulate libvlc_delay directly. Then the end user doesn't have to become aware of the relationship between PTS and the clock - they just get microseconds. But really the main thing is just exposing it and maybe adding a comment someplace so it's clear for intended usage.

Under the hood, the m_AudioPlayer is just wrapping an older version of NAudio here, so I'd taken a brief look at the desired latency approach there. Unfortunately I think it was something set up on the initializer for the WaveOut so it wasn't clear that it could be dynamically set, but maybe newer versions have something better?

As a reference point, I created some debug logs. The first time the callback is called the wait is long, and then it becomes short - much nearer a reasonable audio playback delay. Here's a snippet:

Audio/video sync gap. PTS: 13941455223 Microseconds until play time of that PTS: 1681223
Audio/video sync gap. PTS: 13941515178 Microseconds until play time of that PTS: 50178
Audio/video sync gap. PTS: 13941575189 Microseconds until play time of that PTS: 57189
Audio/video sync gap. PTS: 13941635323 Microseconds until play time of that PTS: 57323

The Thread.Sleep approach - hacky as it is - seems to be working sufficiently well for the time being.

At any rate, I appreciate you taking the time to discuss the approach here. I thought perhaps I was just missing a helpful common pattern, or that perhaps I needed to instead control the desired latency from the VLC side somehow. That link to the OBS approach is quite helpful, too - thanks!

[mfkl]

Would you like to contribute the libvlc_clock binding to LVS 3.x as well as the Delay helper?

[mfkl]

At any rate, I appreciate you taking the time to discuss the approach here. I thought perhaps I was just missing a helpful common pattern, or that perhaps I needed to instead control the desired latency from the VLC side somehow.

It's alright. Happy to help when this is a good question or report like yours. Though to be honest, I haven't run into this problem personally and my knowledge around it is quite limited.

Looking at https://wiki.videolan.org/VLC_command-line_help/, there might be another thing you could try:

--audio-desync=<integer>   Audio desynchronization compensation
          This delays the audio output. The delay must be given in
          milliseconds. This can be handy if you notice a lag between the video
          and the audio.

Not sure but worth a try. new LibVLC("--audio-desync=x")

[jesse-trana]

Thanks @mfkl - I had actually tried that and sadly it didn't seem to work. Appreciate the thought.

[jesse-trana]

Also @mfkl I had missed your earlier comment about contributing when I read the digest. Yes, I'd be interested in that. I was taking a peek at what would make sense there. It looks like the classes in LVS mirror the groupings in libvlc, so I wondered if creating a new Time.cs that internally imported libvlc_clock and then publicly exposed Delay would be a reasonable approach. However, it does look like the libvlc_clock is maybe planning to get imported in this PR already, so maybe it should just get exposed on LibVLC.cs directly. What do you think?

[mfkl]

Indeed, but that PR is for master and you would want to target 3.x (the current stable version).

No need for an extra class, just expose LibVLC.Clock and LibVLC.Delay from the current class, it will be fine. There are already libvlc_** binding in there. Thanks!

[mfkl]

this is tracked here if you wanna give it a go https://code.videolan.org/videolan/LibVLCSharp/-/issues/588

[jesse-trana]

Thanks @mfkl ! I've not done LibVLCSharp development before but I was working on seeing if I could get the environment set up last night so this is timely.

[mfkl]

Feel free to checkout the CI file for the env, and you can also come ask questions on our discord if you need help. Cheers.

[jesse-trana]

Thanks @mfkl - I got stuck on the initial development setup and then had to lay this down. Appreciate your followup on this.