PortAudio / portaudio

PortAudio is a cross-platform, open-source C language library for real-time audio input and output.
Other
1.53k stars 308 forks source link

Win32/WASAPI loopback: no capture progress if no playback #935

Open arkadijs opened 5 months ago

arkadijs commented 5 months ago

Experiencing an issue with capture on WASAPI [Loopback] device not progressing unless any audio playback is running on the corresponding output device. Not progressing: callback not called, synchronous API Read blocks forever, Pa_IsStreamActive() returns 1 - active on a started stream. I have two Windows systems - 7 and 10, where all audio output devices exhibit this behavior - with Microsoft HD Audio driver, and one Windows 10 system - with Realtek driver, where capturing Speakers always works but Headphones requires active playback.

My expectation is to receive silence steady when there is no playback.

If current behavior is expected behavior (of the underlying API), then I suggest this to be documented. If a workaround could be developed then it would be nice to have an explicit platform flag for Pa_OpenStream() to enable it. In some use cases, this could be treated as a feature which may save processing, bandwidth, storage. But, given current inconsistent behavior, capturing loopback is confusing. (looked into reference, wiki, PR #672, and general search for clues)

Test program here.

gcc -o loopback loopback.c -lportaudio -lwinmm -lsetupapi -lole32 -Wall

Tested current PortAudio master HEAD and PR #934. PortAudio built without DirectSound. MSYS2 ucrt64 GCC 14.1.

cmake -B build-release -D CMAKE_BUILD_TYPE=Release -D WIN32=On -D PA_USE_DS=Off
cmake --build build-release -j
cmake --install build-release --prefix /ucrt64

Windows 7, Microsoft HD Audio driver, Core 2 Quad on Asus P5KC:

Device: MME > Microsoft Sound Mapper - Output rate:44100
Device: MME > Speakers (High Definition Audio rate:44100
Device: MME > PHL 222S1-24 (NVIDIA High Defin rate:44100
Device: MME > Digital Audio (S/PDIF) (High De rate:44100
Device: Windows WASAPI > PHL 222S1-24 (NVIDIA High Definition Audio) rate:48000
Device: Windows WASAPI > Speakers (High Definition Audio Device) rate:44100
Device: Windows WASAPI > Digital Audio (S/PDIF) (High Definition Audio Device) rate:44100
Device: Windows WASAPI > PHL 222S1-24 (NVIDIA High Definition Audio) [Loopback] rate:48000 <=======
Device: Windows WASAPI > Speakers (High Definition Audio Device) [Loopback] rate:44100 <=======
Device: Windows WASAPI > Digital Audio (S/PDIF) (High Definition Audio Device) [Loopback] rate:44100 <=======
Device: Windows WDM-KS > Output () rate:48000
Device: Windows WDM-KS > Speakers (HD Audio Speaker) rate:44100
Device: Windows WDM-KS > SPDIF Out (HD Audio SPDIF out) rate:44100
PHL 222S1-24 (NVIDIA High Definition Audio) [Loopback]: trying callback API...
PHL 222S1-24 (NVIDIA High Definition Audio) [Loopback]: no frames pushed
Speakers (High Definition Audio Device) [Loopback]: trying callback API...
Speakers (High Definition Audio Device) [Loopback]: success
Speakers (High Definition Audio Device) [Loopback]: trying blocking API...
Speakers (High Definition Audio Device) [Loopback]: read 512 frames
Speakers (High Definition Audio Device) [Loopback]: read 512 frames
Speakers (High Definition Audio Device) [Loopback]: read 512 frames
Speakers (High Definition Audio Device) [Loopback]: read 512 frames
Speakers (High Definition Audio Device) [Loopback]: read 512 frames
Speakers (High Definition Audio Device) [Loopback]: success
Digital Audio (S/PDIF) (High Definition Audio Device) [Loopback]: trying callback API...
Digital Audio (S/PDIF) (High Definition Audio Device) [Loopback]: no frames pushed

... HDMI playback:

Device: MME > Microsoft Sound Mapper - Output rate:44100
Device: MME > PHL 222S1-24 (NVIDIA High Defin rate:44100
Device: MME > Speakers (High Definition Audio rate:44100
Device: MME > Digital Audio (S/PDIF) (High De rate:44100
Device: Windows WASAPI > PHL 222S1-24 (NVIDIA High Definition Audio) rate:48000
Device: Windows WASAPI > Speakers (High Definition Audio Device) rate:44100
Device: Windows WASAPI > Digital Audio (S/PDIF) (High Definition Audio Device) rate:44100
Device: Windows WASAPI > PHL 222S1-24 (NVIDIA High Definition Audio) [Loopback] rate:48000 <=======
Device: Windows WASAPI > Speakers (High Definition Audio Device) [Loopback] rate:44100 <=======
Device: Windows WASAPI > Digital Audio (S/PDIF) (High Definition Audio Device) [Loopback] rate:44100 <=======
Device: Windows WDM-KS > Output () rate:48000
Device: Windows WDM-KS > Speakers (HD Audio Speaker) rate:44100
Device: Windows WDM-KS > SPDIF Out (HD Audio SPDIF out) rate:44100
PHL 222S1-24 (NVIDIA High Definition Audio) [Loopback]: trying callback API...
PHL 222S1-24 (NVIDIA High Definition Audio) [Loopback]: success
PHL 222S1-24 (NVIDIA High Definition Audio) [Loopback]: trying blocking API...
PHL 222S1-24 (NVIDIA High Definition Audio) [Loopback]: read 512 frames
PHL 222S1-24 (NVIDIA High Definition Audio) [Loopback]: read 512 frames
PHL 222S1-24 (NVIDIA High Definition Audio) [Loopback]: read 512 frames
PHL 222S1-24 (NVIDIA High Definition Audio) [Loopback]: read 512 frames
PHL 222S1-24 (NVIDIA High Definition Audio) [Loopback]: read 512 frames
PHL 222S1-24 (NVIDIA High Definition Audio) [Loopback]: success
Speakers (High Definition Audio Device) [Loopback]: trying callback API...
Speakers (High Definition Audio Device) [Loopback]: no frames pushed
Digital Audio (S/PDIF) (High Definition Audio Device) [Loopback]: trying callback API...
Digital Audio (S/PDIF) (High Definition Audio Device) [Loopback]: no frames pushed

Windows 10, Microsoft HD Audio driver, Intel Core i5 of Haswell generation (two S/PDIF outputs as intended):

Device: MME > Microsoft Sound Mapper - Output rate:44100
Device: MME > Headphones (High Definition Aud rate:44100
Device: MME > Digital Audio (S/PDIF) (High De rate:44100
Device: MME > PHL 222S1 (Intel(R) Display Aud rate:44100
Device: MME > Digital Audio (S/PDIF) (High De rate:44100
Device: Windows WASAPI > Digital Audio (S/PDIF) (High Definition Audio Device) rate:48000
Device: Windows WASAPI > Headphones (High Definition Audio Device) rate:48000
Device: Windows WASAPI > PHL 222S1 (Intel(R) Display Audio) rate:48000
Device: Windows WASAPI > Digital Audio (S/PDIF) (High Definition Audio Device) rate:48000
Device: Windows WASAPI > Digital Audio (S/PDIF) (High Definition Audio Device) [Loopback] rate:48000 <=======
Device: Windows WASAPI > Headphones (High Definition Audio Device) [Loopback] rate:48000 <=======
Device: Windows WASAPI > PHL 222S1 (Intel(R) Display Audio) [Loopback] rate:48000 <=======
Device: Windows WASAPI > Digital Audio (S/PDIF) (High Definition Audio Device) [Loopback] rate:48000 <=======
Device: Windows WDM-KS > SPDIF Out (HD Audio SPDIF out 2) rate:44100
Device: Windows WDM-KS > Headphones (HD Audio Headphone) rate:44100
Device: Windows WDM-KS > SPDIF Out (HD Audio SPDIF out) rate:44100
Device: Windows WDM-KS > Output (Intel(R) Display Audio Output 2) rate:44100
Digital Audio (S/PDIF) (High Definition Audio Device) [Loopback]: trying callback API...
Digital Audio (S/PDIF) (High Definition Audio Device) [Loopback]: no frames pushed
Headphones (High Definition Audio Device) [Loopback]: trying callback API...
Headphones (High Definition Audio Device) [Loopback]: success
Headphones (High Definition Audio Device) [Loopback]: trying blocking API...
Headphones (High Definition Audio Device) [Loopback]: read 512 frames
Headphones (High Definition Audio Device) [Loopback]: read 512 frames
Headphones (High Definition Audio Device) [Loopback]: read 512 frames
Headphones (High Definition Audio Device) [Loopback]: read 512 frames
Headphones (High Definition Audio Device) [Loopback]: read 512 frames
Headphones (High Definition Audio Device) [Loopback]: success
PHL 222S1 (Intel(R) Display Audio) [Loopback]: trying callback API...
PHL 222S1 (Intel(R) Display Audio) [Loopback]: no frames pushed
Digital Audio (S/PDIF) (High Definition Audio Device) [Loopback]: trying callback API...
Digital Audio (S/PDIF) (High Definition Audio Device) [Loopback]: no frames pushed

... HDMI playback:

Device: MME > Microsoft Sound Mapper - Output rate:44100
Device: MME > PHL 222S1 (Intel(R) Display Aud rate:44100
Device: MME > Digital Audio (S/PDIF) (High De rate:44100
Device: MME > Headphones (High Definition Aud rate:44100
Device: MME > Digital Audio (S/PDIF) (High De rate:44100
Device: Windows WASAPI > Digital Audio (S/PDIF) (High Definition Audio Device) rate:48000
Device: Windows WASAPI > Headphones (High Definition Audio Device) rate:48000
Device: Windows WASAPI > PHL 222S1 (Intel(R) Display Audio) rate:48000
Device: Windows WASAPI > Digital Audio (S/PDIF) (High Definition Audio Device) rate:48000
Device: Windows WASAPI > Digital Audio (S/PDIF) (High Definition Audio Device) [Loopback] rate:48000 <=======
Device: Windows WASAPI > Headphones (High Definition Audio Device) [Loopback] rate:48000 <=======
Device: Windows WASAPI > PHL 222S1 (Intel(R) Display Audio) [Loopback] rate:48000 <=======
Device: Windows WASAPI > Digital Audio (S/PDIF) (High Definition Audio Device) [Loopback] rate:48000 <=======
Device: Windows WDM-KS > SPDIF Out (HD Audio SPDIF out 2) rate:44100
Device: Windows WDM-KS > Headphones (HD Audio Headphone) rate:44100
Device: Windows WDM-KS > SPDIF Out (HD Audio SPDIF out) rate:44100
Device: Windows WDM-KS > Output (Intel(R) Display Audio Output 2) rate:44100
Digital Audio (S/PDIF) (High Definition Audio Device) [Loopback]: trying callback API...
Digital Audio (S/PDIF) (High Definition Audio Device) [Loopback]: no frames pushed
Headphones (High Definition Audio Device) [Loopback]: trying callback API...
Headphones (High Definition Audio Device) [Loopback]: no frames pushed
PHL 222S1 (Intel(R) Display Audio) [Loopback]: trying callback API...
PHL 222S1 (Intel(R) Display Audio) [Loopback]: success
PHL 222S1 (Intel(R) Display Audio) [Loopback]: trying blocking API...
PHL 222S1 (Intel(R) Display Audio) [Loopback]: read 512 frames
PHL 222S1 (Intel(R) Display Audio) [Loopback]: read 512 frames
PHL 222S1 (Intel(R) Display Audio) [Loopback]: read 512 frames
PHL 222S1 (Intel(R) Display Audio) [Loopback]: read 512 frames
PHL 222S1 (Intel(R) Display Audio) [Loopback]: read 512 frames
PHL 222S1 (Intel(R) Display Audio) [Loopback]: success
Digital Audio (S/PDIF) (High Definition Audio Device) [Loopback]: trying callback API...
Digital Audio (S/PDIF) (High Definition Audio Device) [Loopback]: no frames pushed

Windows 10, Realtek driver, Ryzen 5600H Lenovo laptop (here test program always call Pa_ReadStream() to demonstrate blocking behavior):

Device: MME > Microsoft Sound Mapper - Input rate:44100
Device: MME > Mikrofon (Realtek(R) Audio) rate:44100
Device: MME > Zestaw mikrofonów (Realtek(R) A rate:44100
Device: MME > Microsoft Sound Mapper - Output rate:44100
Device: MME > Słuchawki (Realtek(R) Audio) rate:44100
Device: MME > Głośniki (Realtek(R) Audio) rate:44100
Device: Windows WASAPI > Głośniki (Realtek(R) Audio) rate:48000
Device: Windows WASAPI > Słuchawki (Realtek(R) Audio) rate:48000
Device: Windows WASAPI > Mikrofon (Realtek(R) Audio) rate:48000
Device: Windows WASAPI > Zestaw mikrofonów (Realtek(R) Audio) rate:48000
Device: Windows WASAPI > Głośniki (Realtek(R) Audio) [Loopback] rate:48000 <=======
Device: Windows WASAPI > Słuchawki (Realtek(R) Audio) [Loopback] rate:48000 <=======
Device: Windows WDM-KS > Speakers 1 (Realtek HD Audio output with HAP) rate:48000
Device: Windows WDM-KS > Speakers 2 (Realtek HD Audio output with HAP) rate:44100
Device: Windows WDM-KS > Głośnik PC (Realtek HD Audio output with HAP) rate:44100
Device: Windows WDM-KS > Mikrofon (Realtek HD Audio Mic input) rate:44100
Device: Windows WDM-KS > Miks stereo (Realtek HD Audio Stereo input) rate:48000
Device: Windows WDM-KS > Zestaw mikrofonów (Realtek HD Audio Mic Array input) rate:44100
Device: Windows WDM-KS > Headphones (Realtek HD Audio 2nd output) rate:44100
Device: Windows WDM-KS > Output (AMD HD Audio HDMI out #0) rate:44100
Device: Windows WDM-KS > Speakers (Nahimic mirroring Wave Speaker) rate:44100
Głośniki (Realtek(R) Audio) [Loopback]: trying callback API...
Głośniki (Realtek(R) Audio) [Loopback]: pushed 512 frames
Głośniki (Realtek(R) Audio) [Loopback]: pushed 512 frames
Głośniki (Realtek(R) Audio) [Loopback]: pushed 512 frames
Głośniki (Realtek(R) Audio) [Loopback]: pushed 512 frames
Głośniki (Realtek(R) Audio) [Loopback]: pushed 512 frames
Głośniki (Realtek(R) Audio) [Loopback]: success
Głośniki (Realtek(R) Audio) [Loopback]: trying blocking API...
Głośniki (Realtek(R) Audio) [Loopback]: read 512 frames
Głośniki (Realtek(R) Audio) [Loopback]: read 512 frames
Głośniki (Realtek(R) Audio) [Loopback]: read 512 frames
Głośniki (Realtek(R) Audio) [Loopback]: read 512 frames
Głośniki (Realtek(R) Audio) [Loopback]: read 512 frames
Głośniki (Realtek(R) Audio) [Loopback]: success
Słuchawki (Realtek(R) Audio) [Loopback]: trying callback API...
Słuchawki (Realtek(R) Audio) [Loopback]: no frames pushed
Słuchawki (Realtek(R) Audio) [Loopback]: trying blocking API...
Słuchawki (Realtek(R) Audio) [Loopback]: read 512 frames <=== unpaused player here
Słuchawki (Realtek(R) Audio) [Loopback]: read 512 frames
Słuchawki (Realtek(R) Audio) [Loopback]: read 512 frames
Słuchawki (Realtek(R) Audio) [Loopback]: read 512 frames
Słuchawki (Realtek(R) Audio) [Loopback]: read 512 frames
Słuchawki (Realtek(R) Audio) [Loopback]: success
dmitrykos commented 4 months ago

@arkadijs thank you for opening this issue.

Indeed, it looks like a WASAPI specific behavior in relation to Loopback devices, i.e. if there are no Shared opened streams delivering audio data to WASAPI engine then Loopback device will not produce any audio data and therefore will timeout.

Besides your example the PA's test pa_loopback fails too PaQa_RunInputOnly() on attempt to capture data from Loopback WASAPI device.

The questions now are:

  1. leave this behavior as is and let client's code decide what to do with time out, whether to fill timed out buffer with silence or fail, and document this behavior in PA WASAPI-specific header or
  2. implement workaround in PA WASAPI to enforce silence if WASAPI Loopback device times out, or
  3. add additional PA WASAPI flag which would enforce filling buffer with silence in case of timeout.
arkadijs commented 4 months ago

(3) looks both helpful and flexible. (1) is good enough for my very specific use-case where not getting any audio (from async API) is actually a feature. This indicates my recorder fell in a wrong state (by tracking Windows Core Audio events) and should better exit. (2) would be consistent with general API behavior though.