mackron / miniaudio

Audio playback and capture library written in C, in a single source file.
https://miniaud.io
Other
4.07k stars 361 forks source link

Incorrect initialization of channel maps using Loopback #660

Closed hgroenenboom closed 1 year ago

hgroenenboom commented 1 year ago

Issue

I'm using WASAPI loopback to capture system audio and it works great most of the time. But specifically when the audiodriver that is captured is SonicID Reference, running at 96khz, the are two types of bugs that occur:

I suspect that this happens because something is going wrong while initializing the channel mapping. The correct channel mapping should be:

INFO:       Channel Map In:         {CHANNEL_FRONT_LEFT CHANNEL_FRONT_RIGHT}
INFO:       Channel Map Out:        {CHANNEL_FRONT_LEFT CHANNEL_FRONT_RIGHT}

Debugging details

I was able to locally fix this issue by hacking the ma_device__post_init_setup function and making sure it always calls ma_channel_map_copy(pDevice->capture.channelMap, pDevice->capture.internalChannelMap, pDevice->capture.channels); when the deviceType is ma_device_type_capture, bypassing all the checks happening in that if statement. Maybe this helps to pin down the problem.

Initialization code used

Code for initialization of the device. (I've tried multiple changes here to fix it as well)

        auto deviceConfig = ma_device_config_init(ma_device_type_loopback);
        deviceConfig.capture.format = ma_format_s16;
        deviceConfig.capture.pDeviceID = NULL;
        deviceConfig.capture.channels = 2;
        deviceConfig.sampleRate = 44100;
        deviceConfig.dataCallback = dataCallbackInternal;
        deviceConfig.pUserData = this;
        if (ma_device_init(NULL, &deviceConfig, &device) != MA_SUCCESS ||
            ma_device_start(&device) != MA_SUCCESS)
        {
            return;
        }

Debug logs

These logs originate from initializing miniaudio twice in my application. both times they initialize with the wrong channel map.

#define MA_DEBUG_OUTPUT
#define MA_DEBUG_OUTPUT: 
Loading symbol: CoInitialize
DEBUG: Loading symbol: CoInitializeEx
DEBUG: Loading symbol: CoUninitialize
DEBUG: Loading symbol: CoCreateInstance
DEBUG: Loading symbol: CoTaskMemFree
DEBUG: Loading symbol: PropVariantClear
DEBUG: Loading symbol: StringFromGUID2
DEBUG: Loading library: user32.dll
DEBUG: Loading symbol: GetForegroundWindow
DEBUG: Loading symbol: GetDesktopWindow
DEBUG: Loading library: advapi32.dll
DEBUG: Loading symbol: RegOpenKeyExA
DEBUG: Loading symbol: RegCloseKey
DEBUG: Loading symbol: RegQueryValueExA
DEBUG: Attempting to initialize WASAPI backend...
DEBUG: Loading library: kernel32.dll
DEBUG: Loading symbol: VerifyVersionInfoW
DEBUG: Loading symbol: VerSetConditionMask
DEBUG: Loading library: avrt.dll
DEBUG: Loading symbol: AvSetMmThreadCharacteristicsA
DEBUG: Loading symbol: AvRevertMmThreadCharacteristics
DEBUG: System Architecture:
DEBUG:   Endian: LE
DEBUG:   SSE2:   YES
DEBUG:   AVX2:   YES
DEBUG:   NEON:   NO
DEBUG: [WASAPI] Trying IAudioClient3_InitializeSharedAudioStream(actualPeriodInFrames=960)
DEBUG:     defaultPeriodInFrames=960
DEBUG:     fundamentalPeriodInFrames=960
DEBUG:     minPeriodInFrames=960
DEBUG:     maxPeriodInFrames=960
DEBUG: [WASAPI] IAudioClient3_InitializeSharedAudioStream failed. Falling back to IAudioClient.
INFO: [WASAPI]
INFO:   Speakers (SoundID Reference Virtual Audio Device) (Capture)
INFO:     Format:      32-bit IEEE Floating Point -> 16-bit Signed Integer
INFO:     Channels:    2 -> 2
INFO:     Sample Rate: 96000 -> 96000
INFO:     Buffer Size: 960*3 (2880)
INFO:     Conversion:
INFO:       Pre Format Conversion:  YES
INFO:       Post Format Conversion: NO
INFO:       Channel Routing:        YES
INFO:       Resampling:             NO
INFO:       Passthrough:            NO
INFO:       Channel Map In:         {CHANNEL_FRONT_LEFT CHANNEL_FRONT_CENTER}
INFO:       Channel Map Out:        {CHANNEL_FRONT_LEFT CHANNEL_FRONT_RIGHT}
DEBUG: [WASAPI] Capture Flags: 1
DEBUG: [WASAPI] Data discontinuity (possible overrun). Attempting recovery. mappedBufferCaptureCap=960

DEBUG: Loading library: ole32.dll
DEBUG: Loading symbol: CoInitialize
DEBUG: Loading symbol: CoInitializeEx
DEBUG: Loading symbol: CoUninitialize
DEBUG: Loading symbol: CoCreateInstance
DEBUG: Loading symbol: CoTaskMemFree
DEBUG: Loading symbol: PropVariantClear
DEBUG: Loading symbol: StringFromGUID2
DEBUG: Loading library: user32.dll
DEBUG: Loading symbol: GetForegroundWindow
DEBUG: Loading symbol: GetDesktopWindow
DEBUG: Loading library: advapi32.dll
DEBUG: Loading symbol: RegOpenKeyExA
DEBUG: Loading symbol: RegCloseKey
DEBUG: Loading symbol: RegQueryValueExA
DEBUG: Attempting to initialize WASAPI backend...
DEBUG: Loading library: kernel32.dll
DEBUG: Loading symbol: VerifyVersionInfoW
DEBUG: Loading symbol: VerSetConditionMask
DEBUG: Loading library: avrt.dll
DEBUG: Loading symbol: AvSetMmThreadCharacteristicsA
DEBUG: Loading symbol: AvRevertMmThreadCharacteristics
DEBUG: System Architecture:
DEBUG:   Endian: LE
DEBUG:   SSE2:   YES
DEBUG:   AVX2:   YES
DEBUG:   NEON:   NO
DEBUG: [WASAPI] Trying IAudioClient3_InitializeSharedAudioStream(actualPeriodInFrames=960)
DEBUG:     defaultPeriodInFrames=960
DEBUG:     fundamentalPeriodInFrames=960
DEBUG:     minPeriodInFrames=960
DEBUG:     maxPeriodInFrames=960
DEBUG: [WASAPI] IAudioClient3_InitializeSharedAudioStream failed. Falling back to IAudioClient.
INFO: [WASAPI]
INFO:   Speakers (SoundID Reference Virtual Audio Device) (Capture)
INFO:     Format:      32-bit IEEE Floating Point -> 16-bit Signed Integer
INFO:     Channels:    2 -> 2
INFO:     Sample Rate: 96000 -> 96000
INFO:     Buffer Size: 960*3 (2880)
INFO:     Conversion:
INFO:       Pre Format Conversion:  YES
INFO:       Post Format Conversion: NO
INFO:       Channel Routing:        YES
INFO:       Resampling:             NO
INFO:       Passthrough:            NO
INFO:       Channel Map In:         {CHANNEL_NONE UNKNOWN}
INFO:       Channel Map Out:        {CHANNEL_FRONT_LEFT CHANNEL_FRONT_RIGHT}
DEBUG: [WASAPI] Capture Flags: 1
DEBUG: [WASAPI] Data discontinuity (possible overrun). Attempting recovery. mappedBufferCaptureCap=960
mackron commented 1 year ago

This is a strange one. So this is only happening with a specific output device? Other ones are working fine? Those input channel maps definitely look suspicious. Are the erroneous channel maps always the same every time you run, or do they tend to change? It's just that I noticed by the looks of your log that it looks like you've initialized two ma_device instances, but they've each got different channel maps.

Are you able to try the simple_playback_sine example with MA_DEBUG_OUTPUT and see if that one reports an erroneous channel map as well?

hgroenenboom commented 1 year ago

It's just that I noticed by the looks of your log that it looks like you've initialized two ma_device instances, but they've each got different channel maps.

sorry this was not clear, but those are two seperate instantiations of miniaudio loopback run after eachother! I'll edit them to be in seperate text fields.

It only happens using loopback. So with simple_playback_sine there are no issues, the channelmapping is then always correct. But if I use simple_lookback.c the channelmappings are inconsistent like in my application. But again, only when using SoundID Reference with 96khz or 88.1khz. I've not been able to reproduce it otherwise.

SoundID Reference is an application that captures audio from an audio driver, applies sound calibration to it, and then presents it as its own output. Do you think it matters that this application already is a virtual audio driver and might be using loopback itself?

The erroneous channelMappings are different every time I run it indeed. Sometimes they are correct as well. I have not been able to find a pattern in it, it seems arbirtrary as far as I can tell.

Here is an example erroneous output from simple_loopback.c:

DEBUG: Loading library: ole32.dll
DEBUG: Loading symbol: CoInitialize
DEBUG: Loading symbol: CoInitializeEx
DEBUG: Loading symbol: CoUninitialize
DEBUG: Loading symbol: CoCreateInstance
DEBUG: Loading symbol: CoTaskMemFree
DEBUG: Loading symbol: PropVariantClear
DEBUG: Loading symbol: StringFromGUID2
DEBUG: Loading library: user32.dll
DEBUG: Loading symbol: GetForegroundWindow
DEBUG: Loading symbol: GetDesktopWindow
DEBUG: Loading library: advapi32.dll
DEBUG: Loading symbol: RegOpenKeyExA
DEBUG: Loading symbol: RegCloseKey
DEBUG: Loading symbol: RegQueryValueExA
DEBUG: Attempting to initialize WASAPI backend...
DEBUG: Loading library: kernel32.dll
DEBUG: Loading symbol: VerifyVersionInfoW
DEBUG: Loading symbol: VerSetConditionMask
DEBUG: Loading library: avrt.dll
DEBUG: Loading symbol: AvSetMmThreadCharacteristicsA
DEBUG: Loading symbol: AvRevertMmThreadCharacteristics
DEBUG: System Architecture:
DEBUG:   Endian: LE
DEBUG:   SSE2:   YES
DEBUG:   AVX2:   YES
DEBUG:   NEON:   NO
INFO: [WASAPI]
INFO:   Speakers (SoundID Reference Virtual Audio Device) (Capture)
INFO:     Format:      32-bit IEEE Floating Point -> 32-bit IEEE Floating Point
INFO:     Channels:    2 -> 2
INFO:     Sample Rate: 44100 -> 44100
INFO:     Buffer Size: 441*3 (1323)
INFO:     Conversion:
INFO:       Pre Format Conversion:  NO
INFO:       Post Format Conversion: NO
INFO:       Channel Routing:        YES
INFO:       Resampling:             NO
INFO:       Passthrough:            NO
INFO:       Channel Map In:         {CHANNEL_FRONT_LEFT CHANNEL_FRONT_CENTER}
INFO:       Channel Map Out:        {CHANNEL_FRONT_LEFT CHANNEL_FRONT_RIGHT}
Press Enter to stop recording...
DEBUG: [WASAPI] Capture Flags: 1
DEBUG: [WASAPI] Data discontinuity (possible overrun). Attempting recovery. mappedBufferCaptureCap=441
mackron commented 1 year ago

It's interesting that the channel map tends to change. I wonder if there's an edge case that's resulting in something not being initialized properly. I've pushed a potential fix to the dev branch. Would you be able to give that a try? It's a bit of a long shot because unfortunately I can't replicate this on my end so I'm sort of having to guess a little bit. If this doesn't work I might need to add some additional debug output to narrow it down a bit.

If you're curious on the fix, details are here in this commit: https://github.com/mackron/miniaudio/commit/e386435af9cbeff76bd8f1123aaf9e54fe330b65

hgroenenboom commented 1 year ago

Amazing that seams to have located the problem! The undefined behavior is now gone, instead there is an initialization error. This is the debug output now when using SoundID Reference on 96khz:

DEBUG: Loading library: ole32.dll
DEBUG: Loading symbol: CoInitialize
DEBUG: Loading symbol: CoInitializeEx
DEBUG: Loading symbol: CoUninitialize
DEBUG: Loading symbol: CoCreateInstance
DEBUG: Loading symbol: CoTaskMemFree
DEBUG: Loading symbol: PropVariantClear
DEBUG: Loading symbol: StringFromGUID2
DEBUG: Loading library: user32.dll
DEBUG: Loading symbol: GetForegroundWindow
DEBUG: Loading symbol: GetDesktopWindow
DEBUG: Loading library: advapi32.dll
DEBUG: Loading symbol: RegOpenKeyExA
DEBUG: Loading symbol: RegCloseKey
DEBUG: Loading symbol: RegQueryValueExA
DEBUG: Attempting to initialize WASAPI backend...
DEBUG: Loading library: kernel32.dll
DEBUG: Loading symbol: VerifyVersionInfoW
DEBUG: Loading symbol: VerSetConditionMask
DEBUG: Loading library: avrt.dll
DEBUG: Loading symbol: AvSetMmThreadCharacteristicsA
DEBUG: Loading symbol: AvRevertMmThreadCharacteristics
DEBUG: System Architecture:
DEBUG:   Endian: LE
DEBUG:   SSE2:   YES
DEBUG:   AVX2:   YES
DEBUG:   NEON:   NO
ERROR: [WASAPI] Native format not supported.
Failed to initialize loopback device.

These are the contents of the pNativeFormat object: image

Thanks for this fix already! This is great for stability. I don't yet understand the complete details of the WAVEFORMATEX struct, but do you think it might also be possible to add support for non WAVEFORMATEXTENSIBLE formats too?

mackron commented 1 year ago

This is interesting. It looks like that error is being thrown from this code:

pData->formatOut = ma_format_from_WAVEFORMATEX((MA_WAVEFORMATEX*)&wf);
if (pData->formatOut == ma_format_unknown) {
    /*
    The format isn't supported. This is almost certainly because the exclusive mode format isn't supported by miniaudio. We need to return MA_SHARE_MODE_NOT_SUPPORTED
    in this case so that the caller can detect it and fall back to shared mode if desired. We should never get here if shared mode was requested, but just for
    completeness we'll check for it and return MA_FORMAT_NOT_SUPPORTED.
    */
    if (shareMode == MA_AUDCLNT_SHAREMODE_EXCLUSIVE) {
        result = MA_SHARE_MODE_NOT_SUPPORTED;
    } else {
        result = MA_FORMAT_NOT_SUPPORTED;
    }

    errorMsg = "[WASAPI] Native format not supported.";
    goto done;
}

Would you be able to show me the content of the wf object, just like you've done with pNativeFormat in your previous post? Maybe if you put a breakpoint on that errorMsg = "[WASAPI] Native format not supported."; line.

mackron commented 1 year ago

I've gone ahead and pushed a potential fix for this most recent error you're encountering. I think the driver of the device you're using might a bug in it. I don't believe the cbSize member should be set to 0. Unless 0 is supposed to mean sizeof(WAVEFORMATEX)? In any case, the fix I just pushed will check for it and handle it.

To answer your earlier question, with these most recent fixes, both WAVEFORMATEX and WAVEFORMATEXTENSIBLE should be supported. WAVEFORMATEXTENSIBLE is the same as WAVEFORMATEX but with some additional members at the end of the struct.

hgroenenboom commented 1 year ago

Thank you so much, the last fix works perfectly!


Here is also the content of the wf object (using https://github.com/mackron/miniaudio/commit/e386435af9cbeff76bd8f1123aaf9e54fe330b65) for completeness.

wf image

static ma_format ma_format_from_WAVEFORMATEX(const MA_WAVEFORMATEX* pWF) image

mackron commented 1 year ago

That's great. Thanks for testing those fixes and the report. I'll get this released soon.