Audio crackles even with high latency, worse on some cores

future-figs commented 5 years ago

Description

Highly audible "pops" and "crackles" occur while a core is running, and can only be remedied by raising the audio latency to excessive levels. This is most prominent by far in mGBA out of all cores tested, but can be reproduced in other cores as well (albeit at lower audio latencies).

Expected behavior

The audio output from Retroarch sounds the same as on the original console, unless the core is unable to run at full speed. The audio is imperceptibly time-stretched to sync with the video and resampled to the desired output sample frequency without distortion. Audio output from the core in use must be buffered for some length of time to accomplish this, inducing latency; however, a buffer of only a few milliseconds should be sufficient for cores which are not very demanding on the hardware in use.

Actual behavior

The audio output from Retroarch is heavily distorted unless the audio latency is set strangely high. The threshold varies by core and by content, and is not always consistent across repeated runs. Mother 3, running on mGBA, usually requires 48 ms minimum but on occasion has required latencies in excess of 80 ms. 20 ms is usually sufficient for most cores, but all mGBA games tested exhibit this audio crackle to some extent at the same latency.

This issue does not affect recorded "video dumps", which contain perfect audio at the emulated system's native sample rate, indicating that the core's output is not at issue here. The distorted audio, however, can be captured with Audacity by monitoring an audio output (e.g. my desktop DAC).

linux audacity

This particularly unlucky run of audio samples shows the full breadth of known failure modes. It affects the left and right channels at the same time. Most of the time, the audio simply "drops out" for a short, but varying length of time. Sometimes the signal appears to have been momentarily "paused", and resumes at the same level it stopped at, but other times it skips either ahead or behind to a completely different level, as if the second chunk of audio has been "time shifted". Additionally, two sections of audio seem to be "spliced" at 27.780---it seems that the audio was abruptly time-shifted without dropping to zero at all. Most of the audio isn't this bad, but these same patterns persist.

I have reproduced this behavior on Linux Mint 19 and Windows 10 on the same machine. The crackle is less severe in Windows, and latency can be lowered a bit compared to Linux, but remains in excess of two frames (32 ms minimum from light testing).

The issue seems to have nothing to do with the core's CPU utilization. The English-language patch of Mother 3 crackles heavily on the main menu screen, making it easy to test with, even though the core can run at approximately 700 FPS in fast-forward mode. Megaman X also exhibited crackling on bsnes-mercury accuracy, but not in any level---only the level select screen, which can also easily run at or above double speed in fast-forward mode.

I have tried changing the output sample rate in RetroArch (48kHz -> 44.1kHz), and I have tried increasing the dynamic audio rate control and the max timing skew (to 3x default values), but without any results. Enabling 0-frame hard GPU sync (ironically) requires additional latency, but I found the worst results (crackling at 64 ms) by reverting to a default config file. This might have been a fluke. My monitor's measured refresh rate converges to 60.000 ± .002 Hz, so my vertical refresh rate is set to 60.000. Changing this value causes more crackles, as expected.

I'm no expert in scheduling theory, but I do know a little bit of it. If memory serves: even with optimal task priorities, the CPU might still miss deadlines, even if it is often idle. I wouldn't be surprised if the audio began to stutter when the system is near full load, or if there was a rare, intermittent gap under moderate load. However, when running such a lightweight core that the system is approximately 90% idle, I would expect there to be enough time to top up the audio buffer at least once per frame, let alone just once every second or third frame.

Also, on the off chance that this is the intended functionality of the audio system, and I'm just setting the latency too low...isn't audio latency just as important as video latency? RetroArch has made pretty amazing progress in the second category recently, and I'd be happy to see audio get some of the same love.

Steps to reproduce the bug

Load content on any core with audio output (mGBA worst known offender, not sure why)
Lower audio latency until crackles present (if necessary)
Test different cores/roms to find threshold latencies

Log file (default config, just let Mother 3 run for a minute; crackles abound)

``` [INFO] [recording] twitch streaming key empty [INFO] RetroArch 1.7.6 (Git 9750719) [INFO] === Build ======================================= Capabilities: MMX MMXEXT SSE1 SSE2 SSE3 SSSE3 SSE4 SSE4.2 AVX AES Built: Feb 6 2019 [INFO] Version: 1.7.6 [INFO] Git: 9750719 [INFO] ================================================= [INFO] Environ SET_PIXEL_FORMAT: RGB565. [INFO] Version of libretro API: 1 [INFO] Compiled against API: 1 [INFO] [Audio]: Set audio input rate to: 29970.03 Hz. [INFO] [Video]: Video @ 960x720 [INFO] [GLX]: GLX_OML_sync_control and GLX_MESA_swap_control supported, using better swap control method... [INFO] [GL]: Found GL context: x [INFO] [GL]: Detecting screen resolution 2944x1080. [INFO] [GLX]: Window manager is Mutter (Muffin). [INFO] [GLX]: X = 0, Y = 0, W = 960, H = 720. [INFO] [GLX]: Found swap function: glXSwapIntervalEXT. [INFO] [GLX]: glXSwapIntervalEXT(1) [INFO] [GL]: Vendor: X.Org, Renderer: AMD TAHITI (DRM 2.50.0, 4.15.0-45-generic, LLVM 7.0.0). [INFO] [GL]: Version: 4.4 (Compatibility Profile) Mesa 18.2.2. [INFO] [GL]: Using resolution 960x720 [INFO] [GL]: Default shader backend found: glsl. [INFO] [Shader driver]: Using GLSL shader backend. [INFO] [GLSL]: Checking GLSL shader support ... [WARN] [GL]: Stock GLSL shaders will be used. [INFO] [GLSL]: Found GLSL vertex shader. [INFO] [GLSL]: Found GLSL fragment shader. [INFO] [GLSL]: Linking GLSL program. [INFO] [GLSL]: Found GLSL vertex shader. [INFO] [GLSL]: Found GLSL fragment shader. [INFO] [GLSL]: Linking GLSL program. [INFO] [GLSL]: Found GLSL vertex shader. [INFO] [GLSL]: Found GLSL fragment shader. [INFO] [GLSL]: Linking GLSL program. [INFO] Setting up menu pipeline shaders for XMB ... [INFO] [GLSL]: Compiling ribbon shader.. [INFO] [GLSL]: Found GLSL vertex shader. [INFO] [GLSL]: Found GLSL fragment shader. [INFO] [GLSL]: Linking GLSL program. [INFO] [GLSL]: Compiling simple ribbon shader.. [INFO] [GLSL]: Found GLSL vertex shader. [INFO] [GLSL]: Found GLSL fragment shader. [INFO] [GLSL]: Linking GLSL program. [INFO] [GLSL]: Compiling snow shader.. [INFO] [GLSL]: Found GLSL vertex shader. [INFO] [GLSL]: Found GLSL fragment shader. [INFO] [GLSL]: Linking GLSL program. [INFO] [GLSL]: Compiling modern snow shader.. [INFO] [GLSL]: Found GLSL vertex shader. [INFO] [GLSL]: Found GLSL fragment shader. [INFO] [GLSL]: Linking GLSL program. [INFO] [GLSL]: Compiling bokeh shader.. [INFO] [GLSL]: Found GLSL vertex shader. [INFO] [GLSL]: Found GLSL fragment shader. [INFO] [GLSL]: Linking GLSL program. [INFO] [GLSL]: Compiling snowflake shader.. [INFO] [GLSL]: Found GLSL vertex shader. [INFO] [GLSL]: Found GLSL fragment shader. [INFO] [GLSL]: Linking GLSL program. [INFO] Resetting shader to defaults ... [INFO] [GL]: Using 4 textures. [INFO] [GL]: Loaded 1 program(s). [INFO] [GL]: Using GL_RGB565 for texture uploads. [INFO] [udev]: Plugged pad: Microsoft X-Box One S pad (1118:746) on port #0. [INFO] [udev]: Pad #0 (/dev/input/event22) supports force feedback. [INFO] [udev]: Pad #0 (/dev/input/event22) supports 16 force feedback effects. [INFO] [Autoconf]: 193 profiles found. [INFO] [autoconf]: selected configuration: /home/lorenzo/.config/retroarch/autoconfig/udev/Microsoft_X-Box_One_S_pad.cfg [INFO] [Joypad]: Found joypad driver: "udev". [INFO] [Font]: Using font rendering backend: freetype. [INFO] [X11]: Suspending screensaver (X11, xdg-screensaver). [INFO] [Video]: Found display server: x11 [INFO] Found shader "/home/lorenzo/.config/retroarch/shaders/DVI-CRT.glslp" [INFO] Found shader "/home/lorenzo/.config/retroarch/shaders/retroarch.glslp" [INFO] [PulseAudio]: Requested 24576 bytes buffer, got 18432. [INFO] [Menu]: Found menu display driver: "gl". [INFO] [Font]: Using font rendering backend: freetype. [INFO] [Font]: Using font rendering backend: freetype. [INFO] [LED]: LED driver = 'null' 0x55bc31346c60 [INFO] [MIDI]: Initializing ... [INFO] [MIDI]: Input disabled. [INFO] [MIDI]: Output disabled. [INFO] [MIDI]: Initialized "alsa" driver. [INFO] SRAM will not be saved. [INFO] Loading history file: [/home/lorenzo/.config/retroarch/content_history.lpl]. [INFO] Loading history file: [/home/lorenzo/.config/retroarch/content_favorites.lpl]. [INFO] Loading history file: [/home/lorenzo/.config/retroarch/content_music_history.lpl]. [INFO] Loading history file: [/home/lorenzo/.config/retroarch/content_video_history.lpl]. [INFO] Loading history file: [/home/lorenzo/.config/retroarch/content_image_history.lpl]. [INFO] [GL]: VSync => on [INFO] [GLX]: glXSwapIntervalEXT(1) [INFO] [PulseAudio]: Unpausing. [INFO] [GL]: VSync => on [INFO] [GLX]: glXSwapIntervalEXT(1) [INFO] [PulseAudio]: Pausing. [INFO] [Font]: Using font rendering backend: freetype. [INFO] [Font]: Using font rendering backend: freetype. [INFO] Updating firmware status for: /usr/lib/x86_64-linux-gnu/libretro/mgba_libretro.so on /media/storage/roms/gba/ [INFO] Using content: /media/storage/roms/gba/Mother 3 (Japan).gba. [INFO] arg #0: retroarch [INFO] arg #1: /media/storage/roms/gba/Mother 3 (Japan).gba [INFO] arg #2: -c [INFO] arg #3: /home/lorenzo/.config/retroarch/retroarch.cfg [INFO] arg #4: -L [INFO] arg #5: /usr/lib/x86_64-linux-gnu/libretro/mgba_libretro.so [INFO] Unloading game.. [INFO] Unloading core.. [INFO] Unloading core symbols.. [INFO] [XINERAMA]: Xinerama version: 1.1. [INFO] [XINERAMA]: Xinerama screens: 2. [INFO] [XINERAMA]: Saved monitor #0. [INFO] [Video]: Does not have enough samples for monitor refresh rate estimation. Requires to run for at least 4096 frames. [INFO] Set config file to : /home/lorenzo/.config/retroarch/retroarch.cfg [INFO] RetroArch 1.7.6 (Git 9750719) [INFO] Redirecting save file to "/media/storage/roms/gba/Mother 3 (Japan).srm". [INFO] Redirecting savestate to "/media/storage/roms/gba/Mother 3 (Japan).state". [INFO] === Build ======================================= Capabilities: MMX MMXEXT SSE1 SSE2 SSE3 SSSE3 SSE4 SSE4.2 AVX AES Built: Feb 6 2019 [INFO] Version: 1.7.6 [INFO] Git: 9750719 [INFO] ================================================= [INFO] Loading dynamic libretro core from: "/usr/lib/x86_64-linux-gnu/libretro/mgba_libretro.so" [INFO] [overrides] no core-specific overrides found at /home/lorenzo/.config/retroarch/config/mGBA/mGBA.cfg. [INFO] [overrides] no content-dir-specific overrides found at /home/lorenzo/.config/retroarch/config/mGBA/gba.cfg. [INFO] [overrides] no game-specific overrides found at /home/lorenzo/.config/retroarch/config/mGBA/Mother 3 (Japan).cfg. [INFO] Shaders: preset directory: /home/lorenzo/.config/retroarch/shaders/presets [INFO] Shaders: no game-specific preset found at /home/lorenzo/.config/retroarch/shaders/presets/mGBA/Mother 3 (Japan).cgp. [INFO] Shaders: no game-specific preset found at /home/lorenzo/.config/retroarch/shaders/presets/mGBA/Mother 3 (Japan).glslp. [INFO] Shaders: no content-dir-specific preset found at /home/lorenzo/.config/retroarch/shaders/presets/mGBA/gba.cgp. [INFO] Shaders: no content-dir-specific preset found at /home/lorenzo/.config/retroarch/shaders/presets/mGBA/gba.glslp. [INFO] Shaders: no core-specific preset found at /home/lorenzo/.config/retroarch/shaders/presets/mGBA/mGBA.cgp. [INFO] Shaders: core-specific shader preset found at /home/lorenzo/.config/retroarch/shaders/presets/mGBA/mGBA.glslp. [INFO] Environ SET_VARIABLES. [INFO] Remaps: remap directory: /home/lorenzo/.config/retroarch/config/remaps [INFO] Remaps: no game-specific remap found at /home/lorenzo/.config/retroarch/config/remaps/mGBA/Mother 3 (Japan).rmp. [INFO] Remaps: no content-dir-specific remap found at /home/lorenzo/.config/retroarch/config/remaps/mGBA/gba.rmp. [INFO] Remaps: core-specific remap found at /home/lorenzo/.config/retroarch/config/remaps/mGBA/mGBA.rmp. [INFO] Redirecting save file to "/media/storage/roms/gba/Mother 3 (Japan).srm". [INFO] Redirecting savestate to "/media/storage/roms/gba/Mother 3 (Japan).state". [INFO] Environ SET_PIXEL_FORMAT: RGB565. [INFO] Environ SET_INPUT_DESCRIPTORS: [INFO] RetroPad, User 1, Button "B (bottom)" => "B" [INFO] RetroPad, User 1, Button "Select" => "Select" [INFO] RetroPad, User 1, Button "Start" => "Start" [INFO] RetroPad, User 1, Button "D-Pad Up" => "Up" [INFO] RetroPad, User 1, Button "D-Pad Down" => "Down" [INFO] RetroPad, User 1, Button "D-Pad Left" => "Left" [INFO] RetroPad, User 1, Button "D-Pad Right" => "Right" [INFO] RetroPad, User 1, Button "A (right)" => "A" [INFO] RetroPad, User 1, Button "L" => "L" [INFO] RetroPad, User 1, Button "R" => "R" [INFO] RetroPad, User 1, Button "L3" => "Darken Solar Sensor" [INFO] RetroPad, User 1, Button "R3" => "Brighten Solar Sensor" [INFO] Environ GET_RUMBLE_INTERFACE. [INFO] Environ GET_LOG_INTERFACE. [INFO] Loading content file: /media/storage/roms/gba/Mother 3 (Japan).gba. [INFO] Did not find a valid content patch. [INFO] CRC32: 0xa323c029 . [INFO] Environ GET_VARIABLE mgba_use_bios: [INFO] ON [INFO] Environ GET_VARIABLE mgba_skip_bios: [INFO] OFF [INFO] Environ GET_VARIABLE mgba_idle_optimization: [INFO] Remove Known [WARN] SYSTEM DIR is empty, assume CONTENT DIR /media/storage/roms/gba/Mother 3 (Japan).gba [INFO] Environ SYSTEM_DIRECTORY: "/media/storage/roms/gba/". [INFO] Environ SET_MEMORY_MAPS. [INFO] ndx flags ptr offset start select disconn len addrspace [INFO] 001 M1A1bc 0x7f90e8405000 00000000 03000000 FF000000 00000000 00008000 [INFO] 002 M1A1bc 0x7f90c2880000 00000000 02000000 FF000000 00000000 00040000 [INFO] 003 M1A1bc 0x7f90ca798000 00000000 0E000000 FFFE0000 00000000 00020000 [INFO] 004 M1A1bC 0x7f908bfff000 00000000 08000000 FE000000 00000000 02000000 [INFO] 005 M1A1bC 0x7f908bfff000 00000000 0A000000 FE000000 00000000 02000000 [INFO] 006 M1A1bC 0x7f908bfff000 00000000 0C000000 FE000000 00000000 02000000 [INFO] 007 M1A1bC 0x7f90c1d30b60 00000000 00000000 FFFFC000 00000000 00004000 [INFO] 008 M1A1bc 0x7f90c84e0000 00000000 06000000 FF000000 00000000 00018000 [INFO] 009 M1A1bc 0x7f90e840dac4 00000000 05000000 FF000000 00000000 00000400 [INFO] 010 M1A1bc 0x7f90e840ded0 00000000 07000000 FF000000 00000000 00000400 [INFO] 011 M1A1bc 0x7f90e840d040 00000000 04000000 FFFFFC00 00000000 00000400 [INFO] Environ SET_SUPPORT_ACHIEVEMENTS: yes. [INFO] Skipping SRAM load.. [INFO] Version of libretro API: 1 [INFO] Compiled against API: 1 [INFO] [Audio]: Set audio input rate to: 32884.61 Hz. [INFO] [Video]: Video @ 720x480 [INFO] [GLX]: GLX_OML_sync_control and GLX_MESA_swap_control supported, using better swap control method... [INFO] [GL]: Found GL context: x [INFO] [GL]: Detecting screen resolution 2944x1080. [INFO] [GLX]: Window manager is Mutter (Muffin). [INFO] [GLX]: X = 0, Y = 0, W = 720, H = 480. [INFO] [GLX]: Found swap function: glXSwapIntervalEXT. [INFO] [GLX]: glXSwapIntervalEXT(1) [INFO] [GL]: Vendor: X.Org, Renderer: AMD TAHITI (DRM 2.50.0, 4.15.0-45-generic, LLVM 7.0.0). [INFO] [GL]: Version: 4.4 (Compatibility Profile) Mesa 18.2.2. [INFO] [GL]: Using resolution 720x480 [INFO] [GL]: Default shader backend found: glsl. [INFO] [Shader driver]: Using GLSL shader backend. [INFO] [GLSL]: Checking GLSL shader support ... [WARN] [GL]: Stock GLSL shaders will be used. [INFO] [GLSL]: Found GLSL vertex shader. [INFO] [GLSL]: Found GLSL fragment shader. [INFO] [GLSL]: Linking GLSL program. [INFO] [GLSL]: Found GLSL vertex shader. [INFO] [GLSL]: Found GLSL fragment shader. [INFO] [GLSL]: Linking GLSL program. [INFO] [GLSL]: Found GLSL vertex shader. [INFO] [GLSL]: Found GLSL fragment shader. [INFO] [GLSL]: Linking GLSL program. [INFO] Setting up menu pipeline shaders for XMB ... [INFO] [GLSL]: Compiling ribbon shader.. [INFO] [GLSL]: Found GLSL vertex shader. [INFO] [GLSL]: Found GLSL fragment shader. [INFO] [GLSL]: Linking GLSL program. [INFO] [GLSL]: Compiling simple ribbon shader.. [INFO] [GLSL]: Found GLSL vertex shader. [INFO] [GLSL]: Found GLSL fragment shader. [INFO] [GLSL]: Linking GLSL program. [INFO] [GLSL]: Compiling snow shader.. [INFO] [GLSL]: Found GLSL vertex shader. [INFO] [GLSL]: Found GLSL fragment shader. [INFO] [GLSL]: Linking GLSL program. [INFO] [GLSL]: Compiling modern snow shader.. [INFO] [GLSL]: Found GLSL vertex shader. [INFO] [GLSL]: Found GLSL fragment shader. [INFO] [GLSL]: Linking GLSL program. [INFO] [GLSL]: Compiling bokeh shader.. [INFO] [GLSL]: Found GLSL vertex shader. [INFO] [GLSL]: Found GLSL fragment shader. [INFO] [GLSL]: Linking GLSL program. [INFO] [GLSL]: Compiling snowflake shader.. [INFO] [GLSL]: Found GLSL vertex shader. [INFO] [GLSL]: Found GLSL fragment shader. [INFO] [GLSL]: Linking GLSL program. [INFO] Resetting shader to defaults ... [INFO] [GL]: Using 4 textures. [INFO] [GL]: Loaded 1 program(s). [INFO] [GL]: Using GL_RGB565 for texture uploads. [INFO] [udev]: Plugged pad: Microsoft X-Box One S pad (1118:746) on port #0. [INFO] [udev]: Pad #0 (/dev/input/event22) supports force feedback. [INFO] [udev]: Pad #0 (/dev/input/event22) supports 16 force feedback effects. [INFO] [Autoconf]: 193 profiles found. [INFO] [autoconf]: selected configuration: /home/lorenzo/.config/retroarch/autoconfig/udev/Microsoft_X-Box_One_S_pad.cfg [INFO] [Joypad]: Found joypad driver: "udev". [INFO] [Font]: Using font rendering backend: freetype. [INFO] [X11]: Suspending screensaver (X11, xdg-screensaver). [INFO] [Video]: Found display server: x11 [INFO] Found shader "/home/lorenzo/.config/retroarch/shaders/DVI-CRT.glslp" [INFO] Found shader "/home/lorenzo/.config/retroarch/shaders/retroarch.glslp" [INFO] [PulseAudio]: Requested 24576 bytes buffer, got 18432. [INFO] [Font]: Using font rendering backend: freetype. [INFO] [Font]: Using font rendering backend: freetype. [INFO] [LED]: LED driver = 'null' 0x55bc31346c60 [INFO] [MIDI]: Initializing ... [INFO] [MIDI]: Input disabled. [INFO] [MIDI]: Output disabled. [INFO] [MIDI]: Initialized "alsa" driver. [INFO] Loading history file: [/home/lorenzo/.config/retroarch/content_history.lpl]. [INFO] Loading history file: [/home/lorenzo/.config/retroarch/content_favorites.lpl]. [INFO] Loading history file: [/home/lorenzo/.config/retroarch/content_music_history.lpl]. [INFO] Loading history file: [/home/lorenzo/.config/retroarch/content_video_history.lpl]. [INFO] Loading history file: [/home/lorenzo/.config/retroarch/content_image_history.lpl]. [INFO] [GL]: VSync => on [INFO] [GLX]: glXSwapIntervalEXT(1) [INFO] [PulseAudio]: Unpausing. [INFO] [Font]: Using font rendering backend: freetype. [INFO] [Font]: Using font rendering backend: freetype. [libretro INFO] GBA Memory: Detected Flash savegame [INFO] [PulseAudio]: Unpausing. [INFO] [PulseAudio]: Unpausing. [INFO] [Config]: Saved new config to "/home/lorenzo/.config/retroarch/retroarch.cfg". [INFO] Saving RAM type #0 to "/media/storage/roms/gba/Mother 3 (Japan).srm". [INFO] Saved successfully to "/media/storage/roms/gba/Mother 3 (Japan).srm". [INFO] Unloading game.. [INFO] [PulseAudio]: Pausing. [INFO] Unloading core.. [INFO] Unloading core symbols.. [INFO] Saved core options file to "/home/lorenzo/.config/retroarch/retroarch-core-options.cfg" [INFO] [XINERAMA]: Xinerama version: 1.1. [INFO] [XINERAMA]: Xinerama screens: 2. [INFO] [XINERAMA]: Saved monitor #0. [INFO] [Video]: Average monitor Hz: 59.665871 Hz. (69.899 % frame time deviation, based on 2048 last samples). [INFO] [Video]: Average monitor Hz: 59.665871 Hz. (69.899 % frame time deviation, based on 2048 last samples). ```

Bisect Results

I have rough memories of this issue since years back, but never dug into it until now. I suspect it's a long-running feature of the audio system, but if there were any major updates to it historically, I would be happy to test out an earlier version. I can give the latest nightly a shot too...maybe tomorrow.

Version/Commit

RetroArch: 1.7.6/9750719

Environment information

OS: Linux Mint 19.1 64-bit, Windows 10 64-bit
Compiler: N/A

ghost commented 5 years ago

mGBA has known audio issues so I would not bother playing with that core when you are investigating these problems. What are your hardware specs? What is your monitor resolution and refresh rate? Always 60 or have you tried others? Have you tried 1.7.5? What audio driver(s) have you tried? Have you tried different sound cards? Some of them just have terrible latency that is unavoidable. Do you experience these issues with the default latency of 64ms? You might need the wasapi audio driver on Windows to go lower than that.

I would also try with fresh configs (move/rename retroarch.cfg and retroarch-core-options.cfg).

future-figs commented 5 years ago

My apologies, it seems that I was a little hasty to pin this issue on Retroarch. Would you believe me if I told you that PulseAudio was the problem all along?

When you mentioned sound cards and drivers, it reminded me about the PA options for fragment sizes. After changing default.pa to contain the line load-module module-udev-detect tsched=no fixed_latency_range=yes, the crackles are gone entirely---I can even set latency to 0(!) in mGBA. Not sure if that'll work for Beetle PSX, but I'm impressed with this performance. Even in fast-forward, the audio has fewer pops.

Changing the fragment sizes in daemon.conf doesn't seem to do anything latency-wise, and the pactl list sinks output is higher than expected, but those are PulseAudio problems.

So, this issue is probably related the timer-based scheduling used by default in Pulse, and also in Windows since Vista (and also MacOS, apparently). RA should work correctly regardless of the audio scheduling method, but for now, reverting to IRQ-based scheduling is a good workaround.

I can test some of these other variations later, but I'm going to be busy for a few days and won't have time to play with audio drivers.

I'm running an FX-6350 @ 4.5 GHz (hardcore stuff I know). I have a 60 Hz LCD display as well as a CRT that can go up to 120 Hz (I still run it at 60 Hz for RetroArch, but I can test at other framerates). I'm using a USB-connected desktop DAC for audio (Behringer UCA202) so there shouldn't be any HDMI audio driver issues.

ghost commented 5 years ago

If you still experience problems I would make sure to try with only a single monitor. Having multiple connected (especially with different refresh rates) has been known to cause audio issues as well for some people.

Ryunam commented 5 years ago

For what it's worth - and I am not sure it is related specifically to the issue that you described here - I have had quite a few audio issues with the libretro core version of mGBA ever since version 0.5.

For example, you cannot set the audio latency below the default rate of 64ms without triggering a lot of hitches, pops, crackles and overall stuttering, while with version 0.4.1 (I have a separate core I compiled from the 0.4.x branch) I can drop audio latency all the way to 16ms on certain titles and around 20-24ms with most content. The audio is probably not as clear or accurate, but audio latency performance is far, far better.

@vanfanel pointed out the same issue here, indicating that it could be due to a bigger than normal audio buffer: https://github.com/libretro/mgba/issues/109

Others are encountering similar problems with Audio Sync on the latest mGBA core: https://github.com/libretro/mgba/issues/134

future-figs commented 5 years ago

I had PA set up slightly wrong earlier—the fragment number/size options do work, and can be used to change the latency globally. I tested a few cores (not rigorously, just one game each) on my system to see how low I can push the latency. Using fragments of 1 ms each:

mGBA 19 ms
bsnes-mercury accuracy: 8 ms
genesis plus GX: 8 ms
Beetle PSX: 10 ms

On top of this, mGBA's behavior is different from the others when there isn't enough buffer. The pops sound louder and have more body (I suspect this is because they last longer), and the entire core slows down if the buffer is small (4-8 ms). Both of these indicate that mGBA's audio implementation is a little screwy.

However, if I run the same tests with timer-based scheduling back on (tsched=yes), I get really different results:

mGBA: 54 ms
bsnes-mercury accuracy: 20 ms
genesis plus GX: 0 ms
beetle PSX: 30 ms

Cores need 2-3 times their normal latency, except for genesis plus GX, which somehow manages with 0. So definitely some core-to-core variation, but I would expect these numbers to match the first set. My intuition tells me that these numbers should all be a little lower, but that might be wrong. So there seems to be two issues: (1) mGBA audio stuff, and (2) RetroArch doesn't play nice with timer-based audio scheduling.

My pet theory is that, because timer-based scheduling will dynamically change the latency/buffer size, the way that audio samples are generated makes a difference. If they come at a steady cadence (genplus?), tsched should hit one underrun, back off by 10 ms, and then hold steady. But if a large number of samples are dumped at once—say, once per frame or so—then maybe tsched keeps tightening latencies while the samples are available, underruns and backs off, then gets greedy again when another batch is available. As a result, you have to tell PulseAudio to start with an overly-large buffer to "bias" it away from underruns.

These results also give me the feeling that RetroArch's audio task has a lower priority than the video-rendering task. It explains why the latencies for bsnes, genplus, and beetle are about half to 3/4 of a frame—it fills the buffer, almost depletes the buffer while rendering a frame, then refills it at the last second. This prioritization makes sense for large latencies (> 1 frame), but once the deadline for audio becomes shorter than video's, we would ideally want these priorities to be swapped. If a core buffers some amount of audio internally before flushing it out to RA in chunks (mGBA?), this would increase the worst-case latency.

Comparing the audio code in genplus vs. mGBA could probably give some insight into both issues. Also, if anybody knows more than I do about RA's task-scheduling paradigm, I would love to hear about it.

It would be interesting to measure the actual audio latency—is 64 ms actually 64 ms?—but I don't have the equipment to do so. Not actually sure how to do that.

@bparker06 I only output to one monitor while I'm using RA, so I don't think that's a factor. I don't physically unplug my LCD, but I use xrandr to disable it (xrandr --output DVI-1 --off) and it turns off on its own, which should put me in the clear.

Tatsuya79 commented 5 years ago

This issue can be relevant: https://github.com/libretro/mgba/issues/134

Tatsuya79 commented 5 years ago

~~We're trying this for mgba:~~ ~~https://github.com/libretro/mgba/pull/140~~

reverted, was causing problems for GB, we just changed the SAMPLES to 512 instead.

future-figs commented 4 years ago

I would like to confirm that this is still an issue. While setting up RA on a new system (much more powerful than my old one), with a freshly-installed operating system (Kubuntu, no longer Mint), I encountered these same crackles, but much worse than I remember. They occurred in all cores I tried (mGBA of course, also bsnes and Genesis Plus GX). Raising the audio latency setting in RA upwards of 100 ms did not resolve it completely, or even nearly. I now have an adaptive-sync monitor, so I am using the "exact content framerate" option, but otherwise my settings are the same.

I was able to attain a working setup using a similar method as before, but my settings from earlier were insufficient; more buffer/latency was required. This backs my perception that the issue has worsened. If anybody has this issue as well, my revised settings are:

`default.pa`

<cut>
.ifexists module-udev-detect.so
load-module module-udev-detect tsched=0  # add tsched option, fixed_latency_range is only for timer-based scheduling
.else
<cut>

`daemon.conf`

<cut>
high-priority = no  ; if we say yes here, PA ignores the realtime options
nice-level = -1  ; nice doesn't matter, using realtime instead

realtime-scheduling = yes
realtime-priority = 5  ; bigger num = higher priority (max 99)

<...>

; 4 frags * 6 ms each = 24 ms total latency
; Can use up 18 ms max before last IRQ comes (18 > 16.667, but might not work at 50 FPS)
; Uses less CPU than 1 ms fragments, possibly more latency
default-fragments = 4
default-fragment-size-msec = 6
<cut>

And set Audio Latency (ms) to 24 in RetroArch's settings.

I don't believe I had realtime scheduling working properly before, but it worked. Now, it won't work with anything less. It still seems that ~17 ms is the magic number here; it must be possible to exhaust an entire frame's worth of audio samples from PA's buffer, while retaining an untouched fragment to trigger an IRQ and refill the buffer on time. For instance, two 12 ms fragments add up to the same total buffer length (i.e. same max latency) as four 6 ms fragments, but it causes dropouts. This would explain the intermittent nature of the problem (start point within a fragment matters), as well as its independence from the computer's horsepower.

Is it possible to log the precise audio-output behavior? (As in, when samples are pushed, how many at once, etc.) This would make the problem easier to dig into.

realnc commented 2 years ago

It turned out that when cores submit audio in multiple steps, audio sync results in crackling with low audio latency settings. Some cores have been fixed now and they result in clean audio even at very low latency settings. The ones I know of that are now fixed are snes9x, mgba, puae, vice, and swanstation.

libretro / RetroArch