[BUG] Higher audio latency after fixes for audio desync

[Stoppedpuma]

Describe the bug Audio latency is roughly twice as much after commits for fixing audio desync issues

To Reproduce gpu-screen-recorder -w screen -f 120 -a "$(pactl get-default-sink).monitor" -cr full -fm cfr -ac opus -k hevc -q very_high -o ~/videos/video.mkv

Expected behavior Lower audio latency

Screenshots "nodesync-build" uses the commits that fixed the audio desync. "desync-build" is the build I used to report #7. the lines show when it appears on video, the waveform shows the audio.

Desktop (please complete the following information):

• X11 or Wayland: Wayland
  • Desktop environment/Window Manager: Sway
• Distro: Debian
• GPU: Radeon RX 6700XT
• Version: Source

• [x] I played the video with the command mpv --no-config video.mp4 (if applicable)
• [ ] I use a laptop with an integrated GPU and a dedicated GPU

[dec05eba]

Just for reference, what is the audio latency like with obs studio when compared in the same way when recording at 120 fps?

[Stoppedpuma]

It's nearly in the middle of the two, I do have obs recording multiple audio tracks (all of which are the same on timing) so I don't know if that affects latency for obs or not.

[dec05eba]

Ok thanks so it might not be possible to sync audio 100% with video if other screen recorders cant do it either, but I know what I can do to improve it at least. I'll test some things when I have time. Just a question, what type of audio output device are you recording with? is it a bluetooth device, speakers, analog headset or what?

[Stoppedpuma]

My output device is my headphones being plugged into the 3.5mm headphone jack on my motherboard. I'm using pipewire with a minimum quantum of 16 and the following allowed clock rates: 44100 48000 88200 96000 176400 192000

OBS audio settings: 48khz Sample rate, Stereo channels, 3 Audio tracks, FFMpeg Opus 192kbps

[dec05eba]

I think one of the reasons for this might be that opus is temporary disabled and with aac a larger audio buffer is used so it might take a tiny bit longer to fill the audio buffer before its saved. But this should be fixable.

[dec05eba]

Hmm I tested this and I cant reproduce it myself (with the command you used). I pretty much dont have any audio/video desync with the latest commit, kdenlive displays the video and audio at the exact same time. I tested with this: https://www.youtube.com/watch?v=ucZl6vQ_8Uo. I tested with taking latency into account but there wasn't any noticable difference. I do have latency if I use my bluetooth device but I get the same latency in all audio recording software with that device and I can fix that by changing latency offset in pavucontrol for the output device.

But I pushed the code with latency compensation to git that you can try.

[Stoppedpuma]

I'll test the new commit now.

Were you using a 120 fps preset on kdenlive during your testing?

[Stoppedpuma]

New commit doesn't seem to help? ~On kdenlive I noticed it was inconsistent between 8-10 moves on the waveform on the new commit (I don't recall it being inconsistent before but I might be wrong)~ (See below the tests), I'm also receiving an error I've not had before on the new commit amdgpu: amdgpu_cs_ctx_create2 failed. (-13)

Comparison tests done in osu!:

compensation (latest commit) desync fix obs

I've rewatched the videos using osu! as a method of comparison instead of the same video you linked, the latest commit does seem to help a bit? The new commit in comparison to the desync fix build seems to be ahead of the previous commit, I can't tell if it's ahead of the video or just me being used to obs latency. I would verify this manually but I seem to have two different applications telling me different things. In case of kdenlive all of them are late, in case of manually scrolling frames in mpv they are early? I am also noticing that the framerate on the latest commit appears to be lower even though it's supposed to be recording at 120 fps? The video file shows 120 fps but it doesn't look like 120 fps when compared to obs and the desync fix build? This might just be from an issue on my end or my eyes tricking me but I'll do more tests spread across more applications, videos, etc for both.

Majority of people aren't really going to notice the delay in less latency sensitive applications and I don't want to burden you with a bunch of different attempts to fix this. If you're fine with how the latency is now then I think it's in an alright enough state.

[Stoppedpuma]

@dec05eba Question, I'm assuming you're not aware of gpu-screen-recorder basically just giving up on trying to output the correct framerate when using stupidly high values such as 240? Should I open another issue for this? I'm able to record just fine at 240 fps on obs.

[dec05eba]

@Stoppedpuma it's likely a hardware limitation with your gpu. are you recording with vaapi and hevc on obs at the same resolution? you can switch codec to record at higher fps. But also when recording at such high fps it wont really work to record with constant framerate if the gpu cant keep up. obs studio doesn't even let me select higher than 120fps with vaapi even though the gpu can handle at least 200 fps.

[Stoppedpuma]

200fps seems to be where things break if I go above it. On OBS I'm able to record 240fps at 1440p, gpu-screen-recorder can't seem to? Both are using vaapi hevc for encoding. Its fine if you consider this out of the scope since it's extremely high, I was just trying to use this high of a framerate for the latency tests.

[dec05eba]

200fps seems to be where things break if I go above it. On OBS I'm able to record 240fps at 1440p, gpu-screen-recorder can't seem to? Both are using vaapi hevc for encoding. Its fine if you consider this out of the scope since it's extremely high, I was just trying to use this high of a framerate for the latency tests.

I tested recording with pure ffmpeg (kmsgrab) and it records at around the same fps as gpu screen recorder so if you are recording with vaapi hevc in obs studio then the fps it reports is not accurate. You would have to step frame by frame in the video it outputs to see that it's really 240fps and not just duplicating frames

[Stoppedpuma]

When going frame by frame in mpv, obs seems to output 240? Maybe I'm misunderstanding this option in obs?

Just to make sure you're seeing the same behaviour on your end, if you record with gpu-screen-recorder at 240fps it just outputs like 15 fps for you as well correct?

[dec05eba]

When going frame by frame in mpv, obs seems to output 240? Maybe I'm misunderstanding this option in obs?

Just to make sure you're seeing the same behaviour on your end, if you record with gpu-screen-recorder at 240fps it just outputs like 15 fps for you as well correct?

it does that if you record with constant frame rate and the encoder cant keep up, the same thing happens in obs studio with constant frame rate. You have to switch to variable framerate. Unfortunately kdenlive cant deal with that. I just tested with obs studio and i got an encoder warning and the actual video framerate is 15 fps but the files reported fps is 240fps

[dec05eba]

I see what you mean with the audio quality now when you recorded with osu. Also the video/audio timing seems weird for you, i'll check if i can reproduce that myself in osu

[Stoppedpuma]

Also the video/audio timing seems weird for you

I was about to run tests on the classic version of osu! because I was wondering if half of the reason the osu! tests seem so different to the rest was just the bass.net latency and it not using my pipewire quantum, I'll upload tests from the old version with a offset I'm 99% sure is correct as well as it using my quantum of 16 (lazer can't do this until this is merged)

[dec05eba]

The video file shows 120 fps but it doesn't look like 120 fps when compared to obs and the desync fix build? This might just be from an issue on my end or my eyes tricking me but I'll do more tests spread across more applications, videos, etc for both.

you are right, you can do frame by frame step with mpv and it repeats some frames. I will revert the commit. But I have read that wlroots has an issue with frame pacing sometimes at around 120 fps so i'll check myself if i can reproduce it

[dec05eba]

I couldn't reproduce that issue. Im not sure if it's because you re-encoded the video or if wlroots introduced that issue

[Stoppedpuma]

None of the videos I posted above were re-encoded, most likely wlroots then.

[dec05eba]

None of the videos I posted above were re-encoded, most likely wlroots then.

oh i assumed so because of the title text, but is that done is osu then? but anyways i reverted the commit that changes the latency. In my own test it doesn't really impact audio latency itself in a better way. I also merged your commit to change default audio bitrate. Thanks :)

[dec05eba]

Also I tested with correct kdenlive preset and higher resolution video/audio sync test (osu is too noisy with audio to analyze it) and the latency for me is the same with obs and gpu screen recorder. I dont know if its maybe an issue with how frames are captured on wlroots instead. I assume you used pipewire with obs. gpu screen recorder uses a different method and I recorded on X11 instead (vsync off, no compositor).

[Stoppedpuma]

is that done is osu then?

Yes, you can add text as a skinnable element.

Tests done on osu! classic:

OBS - Offset Wizard

Offset Wizard Gameplay (uses desync fix commit since you reverted compensation)

I can't record classic gameplay with obs at the moment because an in-game update just happened to break something for wine support, wonderful timing lol. I could technically but it would be a stuttery mess since obs-gamecapture and the wine prefix aren't working correctly together after the update.

the latency for me is the same with obs and gpu screen recorder. I dont know if its maybe an issue with how frames are captured on wlroots instead.

This could honestly just be an issue on my end, be it kdenlive being in a flatpak, misconfiguration on my end, the wlroots frame issue, or me just being used to the audio latency of obs. I also did notice weird things like changing -ac from opus to aac even though opus is disabled and it affecting latency on the kdenlive? Could that be caused by the wlroots issue you're mentioning?

[dec05eba]

maybe, but that offset wizard that you posted looks to have the same latency as obs and gpu screen recorder for me (when I record), I dont think it can get better than that. The obs offset wizard you posted is a broken link. If this is where the latency is at right now then i'll close this issue as its the expected latency when recording.

edit: I dont know why the latency is different in obs for you, but maybe its because of the multiple audio tracks or some audio config that you have changed in obs. But the latency that gpu screen recorder and obs has by default is the same at least.

[Stoppedpuma]

Would your original guess of audio buffers matter still? Also if you're referring to the first few seconds in the obs recording it's most likely the desync from me tabbing in and out.

[dec05eba]

i tested and audio buffer didn't seem to change the audio/video sync. And for the obs recording I meant that its more accurate for your than for me (for me it has the exact same latency as gpu screen recorder), and if that is because you have changed some obs settings or because of multiple audio tracks i dunno.

[dec05eba]

Oh I think I know what it can be... the latency is affected by the devices you record. Which devices did you record in obs studio? some devices can actually make the audio play before the video (my bluetooth does this), so if you have multiple different devices they cancel the latency out. Thats why you need to compare exact same recording in obs and gpu screen recorder with exact same audio device.

[Stoppedpuma]

My multiple tracks contain the following:

Track 1: Desktop and Mic Track 2: Desktop Track 3: Mic

The comparisons are with Track 1 and just desktop audio on gpu-screen-recorder. Both are using the same output device.

[dec05eba]

i dont know if you can just compare the track alone. Obs might have some track syncing going on. It can also affect how pipewire itself behaves when you record from multiple devices at once.

[Stoppedpuma]

Wait I might be stupid and possibly know what it is, gpu-screen-recorder records at 44.1khz, my obs profile records at 48khz. Sample rates affect pipewire latency so maybe this could be it?

[dec05eba]

gpu screen recorder records at 48khz

[Stoppedpuma]

I must have misread 44.1khz some where then, I'll do single track testing.

I still do have the amdgpu: amdgpu_cs_ctx_create2 failed. (-13) error from latest commit, should I open this as another issue or is this a quick fix?

[dec05eba]

https://gitlab.freedesktop.org/wlroots/wlroots/-/issues/3543#note_1661557 it's this warning, it's harmless. But that shouldn't appear if you run gpu screen recorder after you install it with sudo ./install.sh. Are you instead building it and then running it from the same directory?

[dec05eba]

I must have misread 44.1khz some where then, I'll do single track testing.

it might not just be single track but other obs settings you have changed. I dont know if its worth testing without resetting obs to default. But its ok, you dont have to go through the trouble for that. I believe from the video you posted that it's not an issue in gpu screen recorder as it matches obs for me.

But oh I have one question. When I tested with obs studio and gpu screen recorder I used the audio output capture (pulseaudio) with a pipewire server (gpu screen recorder also uses pulseaudio client code). Did you also do that in obs studio? or did you use a pipewire audio output capture (i dont even have such an option in my obs)?

[Stoppedpuma]

But that shouldn't appear if you run gpu screen recorder after you install it with sudo ./install.sh. Are you instead building it and then running it from the same directory?

Didn't even think of that being a reason, fixed after install.

Did you also do that in obs studio?

I didn't add the audio as a source, I chose my output in settings so whichever one that uses?

Maybe these values are why I'm noticing a difference?

[dec05eba]

that only seems to make a difference in microseconds so I dont think so. But obs syncs well for me now as well, I dont know what I changed.. But I pushed a change to gpu screen recorder. Can you try it? if it doesn't work well then fuck it

[Stoppedpuma]

I haven't the slighest fucking idea what you did either but it's basically on the same frame now!

[dec05eba]

I added a hardcoded value that it shifts the audio by. Hey, if it works it works lol

[Stoppedpuma]

The yellow is where the audio outputs for all the recordings

Correction: the red one titled desync fix is acutally the commit prior to the latest

[dec05eba]

Thanks for testing. I'll go with this until somebody complains that it broke audio sync for them xd