[BUG] Audio desyncs from video when switching sinks

YozoraWolf commented 7 months ago

Describe the bug I was testing the flatpak version (provided by @dec05eba) which gratefully fixed the audio bug when using pulseaudio, so I decided to use two sinks and see the behavior. Everything seems to sound ok, but as soon as I switch sinks (as seen on the video) the sound seems to overlap over the other and then cause major synchronization issues.

To Reproduce The GPU Screen Recorder command you ran or if you used the GUI version then describe which options you used.

Use two sinks (or two sound outputs)

Run:

flatpak run --command=gpu-screen-recorder com.dec05eba.gpu_screen_recorder -w screen -fm cfr -f 60 -a "alsa_output.pci-0000_06_00.6.HiFi__hw_Generic__sink.monitor|bluez_sink.C0_91_B9_DE_7C_AE.a2dp_sink.monitor" -ac aac -k hevc -c mp4 -r 60 -o /home/wolf/Videos

Go to sound settings and switch between those two sinks
Once you save the recording and watch it, you'll notice the desync bug right away when you switch sinks on the video

Expected behavior

The audio should remain in sync even if you switch between sinks. It is my understanding that unless you separate the sinks using the option -a , then by separating them using | (pipe), should merge them into a single track, thus monitoring both sinks and keeping the audio on sync throughout the video.

Screenshots

https://github.com/dec05eba/gpu-screen-recorder-issues/assets/5296711/e9daffc6-ebc1-4add-ac0e-9d73e1d9ad3f

Desktop (please complete the following information):

X11 or Wayland: X11
- Desktop environment/Window Manager: GNOME3
Distro: Linux Mint 21.3 Cinnamon
GPU: NVIDIA Corporation GA106M [GeForce RTX 3060 Mobile / Max-Q]
Version: flatpak (provided by @dec05eba, soon to go live)

Additional context

mpv --no-config output:

wolf@wolf-mint:~/Videos/Sound$ mpv --no-config Replay_2024-04-11_18-49-14.mp4 
 (+) Video --vid=1 (*) (hevc 1920x1080 60.000fps)
 (+) Audio --aid=1 (*) (aac 2ch 48000Hz)
AO: [pulse] 48000Hz stereo 2ch float
VO: [gpu] 1920x1080 yuv420p
Invalid audio PTS: 1.022333 -> 1.503896
Invalid audio PTS: 1.525229 -> 2.002854
AV: 00:00:21 / 00:00:49 (42%) A-V:  0.000
Invalid audio PTS: 19.841812 -> 20.319396
Invalid audio PTS: 20.340729 -> 20.822271
Invalid audio PTS: 20.843604 -> 21.321208
Invalid audio PTS: 21.342542 -> 21.649771
AV: 00:00:21 / 00:00:49 (43%) A-V:  0.000

Audio/Video desynchronisation detected! Possible reasons include too slow
hardware, temporary CPU spikes, broken drivers, and broken files. Audio
position will not match to the video (see A-V status field).

AV: 00:00:17 / 00:00:49 (34%) A-V:  0.000
Invalid audio PTS: 15.838500 -> 16.315958
Invalid audio PTS: 16.337292 -> 16.814833
Invalid audio PTS: 16.836167 -> 17.317687
Invalid audio PTS: 17.339021 -> 17.816521
Invalid audio PTS: 17.837854 -> 18.319562
Invalid audio PTS: 18.340896 -> 18.818500
Invalid audio PTS: 18.839833 -> 19.317500
Invalid audio PTS: 19.338833 -> 19.820479
Invalid audio PTS: 19.841812 -> 20.319396
Invalid audio PTS: 20.340729 -> 20.822271
Invalid audio PTS: 20.843604 -> 21.321208
Invalid audio PTS: 21.342542 -> 21.649771
AV: 00:00:16 / 00:00:49 (32%) A-V:  0.000
Invalid audio PTS: 15.838500 -> 16.315958
Invalid audio PTS: 16.337292 -> 16.814833
Invalid audio PTS: 16.836167 -> 17.317687
Invalid audio PTS: 17.339021 -> 17.816521
Invalid audio PTS: 17.837854 -> 18.319562
Invalid audio PTS: 18.340896 -> 18.818500
Invalid audio PTS: 18.839833 -> 19.317500
Invalid audio PTS: 19.338833 -> 19.820479
Invalid audio PTS: 19.841812 -> 20.319396
Invalid audio PTS: 20.340729 -> 20.822271
AV: 00:00:14 / 00:00:49 (30%) A-V: -0.001
Invalid audio PTS: 15.339604 -> 15.817167
AV: 00:00:14 / 00:00:49 (30%) A-V:  0.475
Invalid audio PTS: 15.838500 -> 16.315958

Audio/Video desynchronisation detected! Possible reasons include too slow
hardware, temporary CPU spikes, broken drivers, and broken files. Audio
position will not match to the video (see A-V status field).

AV: 00:00:15 / 00:00:49 (30%) A-V:  0.939
Invalid audio PTS: 16.337292 -> 16.814833
AV: 00:00:15 / 00:00:49 (30%) A-V:  1.404
Invalid audio PTS: 16.836167 -> 17.317687
AV: 00:00:15 / 00:00:49 (30%) A-V:  1.865
Invalid audio PTS: 17.339021 -> 17.816521
AV: 00:00:15 / 00:00:49 (31%) A-V:  2.329 ct:  0.059
Invalid audio PTS: 17.837854 -> 18.319562
AV: 00:00:15 / 00:00:49 (31%) A-V:  2.798 ct:  0.072
Invalid audio PTS: 18.340896 -> 18.818500
AV: 00:00:15 / 00:00:49 (31%) A-V:  3.255 ct:  0.092
Invalid audio PTS: 18.839833 -> 19.317500
AV: 00:00:15 / 00:00:49 (31%) A-V:  3.720 ct:  0.106
Invalid audio PTS: 19.338833 -> 19.820479
AV: 00:00:15 / 00:00:49 (31%) A-V:  4.188 ct:  0.119
Invalid audio PTS: 19.841812 -> 20.319396
AV: 00:00:15 / 00:00:49 (31%) A-V:  4.645 ct:  0.139
Invalid audio PTS: 20.340729 -> 20.822271
AV: 00:00:15 / 00:00:49 (31%) A-V:  5.114 ct:  0.152
Invalid audio PTS: 20.843604 -> 21.321208
Invalid audio PTS: 21.342542 -> 21.649771
AV: 00:00:33 / 00:00:49 (68%) A-V:  0.004 ct:  6.050
Invalid audio PTS: 34.090521 -> 34.572833
AV: 00:00:33 / 00:00:49 (68%) A-V:  0.480 ct:  6.057
Invalid audio PTS: 34.594167 -> 35.071792
AV: 00:00:33 / 00:00:49 (68%) A-V:  0.951 ct:  6.064
Invalid audio PTS: 35.093125 -> 35.570687
AV: 00:00:33 / 00:00:49 (68%) A-V:  1.408 ct:  6.084
Invalid audio PTS: 35.592021 -> 36.073771
AV: 00:00:33 / 00:00:49 (68%) A-V:  1.877 ct:  6.097
Invalid audio PTS: 36.095104 -> 36.572771
AV: 00:00:33 / 00:00:49 (68%) A-V:  2.341 ct:  6.110
Invalid audio PTS: 36.594104 -> 37.071667
AV: 00:00:33 / 00:00:49 (68%) A-V:  2.798 ct:  6.130
Invalid audio PTS: 37.093000 -> 37.574542
AV: 00:00:33 / 00:00:49 (69%) A-V:  3.267 ct:  6.144
Invalid audio PTS: 37.595875 -> 38.073500
Invalid audio PTS: 38.094833 -> 38.576563
AV: 00:00:34 / 00:00:49 (69%) A-V:  4.213 ct:  6.157
Invalid audio PTS: 38.597896 -> 39.075542
AV: 00:00:34 / 00:00:49 (69%) A-V:  4.670 ct:  6.177
Invalid audio PTS: 39.096875 -> 39.574438
AV: 00:00:34 / 00:00:49 (69%) A-V:  5.135 ct:  6.190
Invalid audio PTS: 39.595771 -> 40.077521
AV: 00:00:34 / 00:00:49 (69%) A-V:  5.603 ct:  6.204
Invalid audio PTS: 40.098854 -> 40.576333
AV: 00:00:34 / 00:00:49 (69%) A-V:  6.060 ct:  6.224
Invalid audio PTS: 40.597667 -> 41.079229
AV: 00:00:34 / 00:00:49 (69%) A-V:  6.529 ct:  6.237
Invalid audio PTS: 41.100562 -> 41.578146
AV: 00:00:34 / 00:00:49 (69%) A-V:  6.993 ct:  6.250
Invalid audio PTS: 41.599479 -> 42.077062
AV: 00:00:34 / 00:00:49 (69%) A-V:  7.450 ct:  6.270
Invalid audio PTS: 42.098396 -> 42.580062
AV: 00:00:34 / 00:00:49 (69%) A-V:  7.919 ct:  6.284
Invalid audio PTS: 42.601396 -> 43.078896
AV: 00:00:34 / 00:00:49 (69%) A-V:  8.383 ct:  6.297
Invalid audio PTS: 43.100229 -> 43.581792
AV: 00:00:34 / 00:00:49 (69%) A-V:  8.851 ct:  6.310
Invalid audio PTS: 43.603125 -> 43.963062
(Paused) AV: 00:00:45 / 00:00:49 (92%) A-V:  0.000 ct: 10.064

Exiting... (Quit)

This may be linked to #7 . But I decided to make it a child issue as it is reproduced in another way.

[x] I played the video with the command mpv --no-config video.mp4 (if applicable)
[x] I use a laptop with an integrated GPU and a dedicated GPU

dec05eba commented 7 months ago

I can reproduce it on linux mint but not arch linux. It also doesn't seem to be related to mixing audio or using multiple audio sources. It happens when recording from certain audio devices and when they go to sleep when not in use and then waking up. I dont know a good solution for this now.

YozoraWolf commented 7 months ago

I just ran one of the tests:

Recording only one sink and switching between them:

https://github.com/dec05eba/gpu-screen-recorder-issues/assets/5296711/62c1ea37-f410-4666-b8b5-0d6d964434db

Only silence when switching sinks (expected behavior)

I can reproduce it on linux mint but not arch linux. It also doesn't seem to be related to mixing audio or using multiple audio sources. It happens when recording from certain audio devices and when they go to sleep when not in use and then waking up. I dont know a good solution for this now, ffmpeg expects audio all the time so if the audio device doesn't give any data.. it starts skipping :/

I see, would this be a behavior tied to pulseaudio perhaps? Don't know if it can happen with pipewire.

This is not a big issue as my GSR Scripts repo originally restarted replay when default sink changed (a version I never pushed to public, it now just detects new sinks and pipes them one after the other). If anything I think I'll make it switch to its previous behavior given that audio desync is no longer an issue.

I will continue to monitor the issue and see if I brainstorm up some ideas later. We can keep this on hold.

And again, thanks for the support too!

dec05eba commented 7 months ago

I think it can happen on pipewire as well but its much less common. I'll look at this issue tomorrow.

dec05eba commented 7 months ago

Confirmed to be have been fixed, closing. Thanks!

dec05eba / gpu-screen-recorder-issues

[BUG] Audio desyncs from video when switching sinks #10