video-audio drift

[hholst80]

I noticed that I get audio/video drift (they do not sync tight) on wf-recorder. I noticed I have the same issue with wl-screenrec.

Is there any remedies I can take to avoid the lag/drift that appears? It usually takes 5min+ for the drift to be noticeable and I think it says <1sec. But even a few hundred ms in drift starts to be noticeable and irritating. At one second I'd say the recording is broken.

I am using pulse audio and the options:

wl-screenrec --low-power=off --encode-resolution 1920x1080 --audio 

Output #0, mp4, to 'screenrecord.mp4':
  Metadata:
    encoder         : Lavf60.16.100
  Stream #0:0: Video: h264 (High) (avc1 / 0x31637661), vaapi, 1920x1080, q=2-31, 40000 kb/s, 90k tbn
  Stream #0:1: Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz, stereo, fltp, 128 kb/s

https://github.com/ammen99/wf-recorder/issues/241

[russelltg]

Thanks for the report. I can't seem to repro (made a 7:30 video). Is the audio behind or ahead?

Currently wl-screenrec assumes that it's being fed exactly sample_rate samples per second. If this isn't the case, the audio could end up behind the video.

It's also possible that the video and audio come from different clocks, in which case we will need something like -async 1, as noted on the wl-recorder issue.

[hholst80]

Is the audio behind or ahead?

I think it actually has been both ways, but when I noticed this time the audio was ahead of the video. My recording locally has the same drift as the YouTube asset below:

https://youtu.be/vsXXgenUCDY?t=1343

I suspect the drift is even more than one second at this point.

Currently wl-screenrec assumes that it's being fed exactly sample_rate samples per second.

I do not know the internals of how this works, but what happens if there is a dropped chunk of audio or video? Is there any compensation?

[russelltg]

Video is timestamped using the timestamp from the compositor (which compositor are you using btw?) And audio is timestamped using how many samples have been received from the beginning of the stream, so audio isn't robust to dropout. This isn't ideal and I'll post a patch here that you can test so see if that's the issue.

[hholst80]

which compositor are you using btw?

[russelltg]

Both sway and Hyprland (correctly, imo) sample from the monotonic clock, so that shouldn't be why I can't repro.

Can I assume you're using the pipewire pulseaudio implementation?

[hholst80]

No pipewire, only pulse audio.

[russelltg]

What does pacman -Q | grep pulse give

[hholst80]

I saw that the 'neofetch' command was from my desktop, but I have the exact same setup (famous last words, right) on both the laptop and the workstation.

Anyways:

[I] root@trump ~
# pacman -Q | grep pulse
libpulse 16.1-7
pulseaudio 16.1-7
pulseaudio-jack 16.1-7
[I] root@trump ~
# 

[russelltg]

Can you attempt installing pipewire-pulse (which most distros are shipping by default) instead of pulseaudio and see if it still repros?

(note that I still consider this a bug, but might be a ok workaround for now)

[hholst80]

I managed to install pipewire again (changed Window mananger to Hyprland) hoping that 1) it will not break firefox 2) it will fix wl-screenrec. ;-)

[hholst80]

I installed hyprland and the pipewire stack again. It seems to work fine with Firefox so far.

As an added bonus, I just recorder 45 minutes of minecraft with my son and we had no audio/video lag in the recording.

[russelltg]

Glad you have a workaround! I'll look into adding a clock filter+resampler. Fixing this will not be trivial so may not be soon

[hholst80]

Closing this not to bulk up on your tidy Issue page. Drift when it happens is very irritating so any fundamental work to protect against this would be great to have.

Thanks for the great work you put into this fine tool!

[russelltg]

Appreciate the sentiment but it is an issue and I'd like to keep it open for anyone else who runs into it and so I don't forget it exists.