Support for XAudio2 via the Wine-4.2 compatibility layer

I previously made the post about Obtaining Ambisonic output in Windows https://github.com/kcat/openal-soft/issues/984. I started getting off topic with the goal of supporting XAudio2 through cross-compiling the Wine-4.2 DLLs (XAudio2_7.dll etc.). This seems like a project that will take a while to bring to fruition if it ends up working at all. I'm creating this area to document how to cross-compile the Wine-4.2 XAudio2 DLLs and how to compile a JACK-backend-capable version of libopenal-1.dll that can interface with those cross-compiled Wine DLLs.

Commands for crosscompiling wine-4.2 with i686-mingw-w64 with OpenAL Run MSYS2 MINGW32

To cross-compile WINE 4.2 DLLs to Windows using i686-w64-mingw32 in MSYS2 on Windows, follow these steps:

Install MSYS2 https://www.msys2.org/ run MSYS2 MINGW32 from the start menu and update the package database within msys2: pacman -Sy
Install the i686-w64-mingw32 toolchain: pacman -S mingw-w64-i686-toolchain
Download wine-4.2.tar.xz and extract WINE 4.2 source code: wine-4.2.tar.xz
Navigate to the WINE source code directory: cd wine-4.2
Install required packages: Run the command "pacman -S git make flex bison" to install required packages for building WINE.
Install openal-soft dependency: pacman -S mingw-w64-i686-openal 6a. Additionally one may copy OpenAL32.lib from a compiled MSVS version of OpenAL Soft into C;\msys64\mingw32\lib continue with step 7a if you take this path.
Configure WINE for cross-compilation: ac_cv_lib_soname_openal=./lib/libopenal.dll.a ./configure --prefix=/usr --host=i686-w64-mingw32 --without-freetype 7a. Alternatively configure WINE for cross-compilation: ac_cv_lib_soname_openal=./lib/OpenAL32.lib ./configure --prefix=/usr --host=i686-w64-mingw32 --without-freetype
Cross-compile WINE DLLs to Windows: make cross_compile=mingw32 -k -j4 --keep-going

This will take an hour to compile the tools. Afterwards you'll have many cross-compiled DLLs XAudio2_7.dll etc. There will also be a testing program in C:/msys64/home/username/wine-4.2/dlls/xaudio2_7/tests/ that can be used to test the functionality of XAudio2_7.dll. It errors out when using a Visual Studio compiled version of OpenAL Soft 1.23.1 (renamed to libopenal-1.dll) ~~but completes tests when used with the MSYS version of libopenal-1.dll.~~ EDIT: The tests run correctly because they are defaulting to the official Microsoft XAudio2 files.

My built files and source code used can be found here: wine-4.2-xact-bin+src.tar.xz.zip

I'm making progress towards the XAudio2 goal with the old WINE 4.2 libraries. I have successfully compiled them in MSYS2 MINGW-W32 using both the Visual Studio LIB file and the linux SO file for linking libraries. ~~They complete the xaudio2-tests with the included libopenal-1.dll build of the 1.23.1 version.~~ EDIT: The tests run correctly because they are defaulting to the official Microsoft XAudio2 files. When I use a natively (Visual Studio 2019) compiled OpenAL32.DLL version of the same build I've been using previously (commit OpenAL Soft 1.23.1 5b6e0df), the execution crashes with an Access Violation.

So I've set out to crosscompile another libopenal-1.dll using msys2 with JACK enabled.

I have now successfully cross-compiled msys-libopenal-1.dll for use with the cross-comipled Wine-4.2 XAudio2 DLLs. My instructions for doing so are below:

To cross-compile OpenAL Soft 1.23.1 with JACK support to Windows using i686-w64-mingw32 in MSYS2 on Windows, follow these steps:

On Windows: Download and Install MSYS2 https://www.msys2.org/ run MSYS2 MINGW32 from the start menu.
Within MSYS2: Update the package database, Run the command: pacman -Sy
Within MSYS2: Install the i686-w64-mingw32 toolchain: pacman -S mingw-w64-i686-toolchain
Within MSYS2: Install required packages: Run the command: pacman -S git make flex bison
On Windows: Download and install https://github.com/jackaudio/jack2-releases/releases/download/v1.9.22/jack2-win64-v1.9.22.exe
On Windows: Copy folder %PROGRAMFILES%\JACK2\include\jack to C:\msys64\mingw32\include
On Windows: Copy file %PROGRAMFILES%\JACK2\lib32\libjack.lib to C:\msys64\mingw32\lib
On Windows: Download and extract https://www.openal-soft.org/openal-releases/openal-soft-1.23.1.tar.bz2 to C:\msys64\home\username\openal-soft
Within MSYS2: Navigate to the openal-soft folder: cd openal-soft
Within MSYS2: To configure the project, Run the command: cmake ./openal-soft-1.23.1/ -DHOST=i686-w64-mingw32 -DBUILD_SHARED_LIBS=ON
On Windows: To disable uneccessary backands edit C:\msys64\home\username\openal-soft\CMakeCache.txt to set disable all backends except for JACK and require JACK backend.
On Windows: To show cmake where the include directory is for JACK edit: C:\msys64\home\username\openal-soft\CMakeCache.txt to set JACK_INCLUDE_DIR:PATH=/include/jack
On Windows: To show cmake where the library file is for JACK edit C:\msys64\home\username\openal-soft\CMakeCache.txt to set JACK_LIBRARY:FILEPATH=/mingw32/lib/libjack.lib
Within MSYS2: configure the project: cmake ./openal-soft-1.23.1/ -DHOST=i686-w64-mingw32 -DBUILD_SHARED_LIBS=ON
Within MSYS2: to re-configure the project with jack linked, Run the command: cmake ./openal-soft-1.23.1/ -DHOST=i686-w64-mingw32 -DBUILD_SHARED_LIBS=ON
Within MSYS2: to build msys-openal-1.dll, Run the command: cmake --build . INFO: msys-openal-1.dll has a few dependencies: libgcc_s_dw2-1.dll, libstdc++-6.dll and libwinpthread-1.dll from C:\msys64\mingw32\bin and msys-openal-1.dll should be renamed libopenal-1.dll for use with XAudio2_7.dll etc.

My built files and source code used can be found here: libopenal-1-1.23.1+jack-bin+src.zip

Programs still crash with an access violation, back to the drawing board.

When running tests:

Using native Microsoft XACT DLL files.

xaudio2_7_test.exe xaudio2.c:204: Test failed: Got unexpected channel mask: 0x63f 199c:xaudio2: 1089 tests executed (0 marked as todo, 1 failure), 0 skipped.

MSVS OpenAL32.lib linked XACT DLLs with no libopenal-1.dll present

xaudio2_7_test.exe xaudio2.c:1010: Test failed: xapofx1_1.dll: CreateFX({a90bc001-e897-e897-7439-43 5500000001}) failed: 80040154 xaudio2.c:1010: Test failed: xapofx1_1.dll: CreateFX({a90bc001-e897-e897-7439-43 5500000003}) failed: 80040154 xaudio2.c:1010: Test failed: xapofx1_1.dll: CreateFX({c4137916-2be1-46fd-8599-44 1536f49856}) failed: 80040154 xaudio2.c:1010: Test failed: xapofx1_1.dll: CreateFX({5039d740-f736-449a-84d3-a5 6202557b87}) failed: 80040154 xaudio2.c:1028: Test failed: xapofx1_1.dll: CreateFX({f5ca7b34-8055-42c0-b836-21 6129eb7e30}) failed: 8007007e xaudio2.c:1039: Test failed: xapofx1_1.dll: CreateFX({629cf0de-3ecc-41e7-9926-f7 e43eebec51}) failed: 8007007e xaudio2.c:1010: Test failed: xapofx1_2.dll: CreateFX({a90bc001-e897-e897-7439-43 5500000001}) failed: 80040154 xaudio2.c:1010: Test failed: xapofx1_2.dll: CreateFX({a90bc001-e897-e897-7439-43 5500000003}) failed: 80040154 xaudio2.c:1010: Test failed: xapofx1_2.dll: CreateFX({c4137916-2be1-46fd-8599-44 1536f49856}) failed: 80040154 xaudio2.c:1010: Test failed: xapofx1_2.dll: CreateFX({5039d740-f736-449a-84d3-a5 6202557b87}) failed: 80040154 xaudio2.c:1028: Test failed: xapofx1_2.dll: CreateFX({e180344b-ac83-4483-959e-18 a5c56a5e19}) failed: 8007007e xaudio2.c:1039: Test failed: xapofx1_2.dll: CreateFX({9cab402c-1d37-44b4-886d-fa 4f36170a4c}) failed: 8007007e xaudio2.c:1010: Test failed: xapofx1_3.dll: CreateFX({a90bc001-e897-e897-7439-43 5500000001}) failed: 80040154 xaudio2.c:1010: Test failed: xapofx1_3.dll: CreateFX({a90bc001-e897-e897-7439-43 5500000003}) failed: 80040154 xaudio2.c:1010: Test failed: xapofx1_3.dll: CreateFX({c4137916-2be1-46fd-8599-44 1536f49856}) failed: 80040154 xaudio2.c:1010: Test failed: xapofx1_3.dll: CreateFX({5039d740-f736-449a-84d3-a5 6202557b87}) failed: 80040154 xaudio2.c:1028: Test failed: xapofx1_3.dll: CreateFX({c7338b95-52b8-4542-aa79-42 eb016c8c1c}) failed: 8007007e xaudio2.c:1039: Test failed: xapofx1_3.dll: CreateFX({8bb7778b-645b-4475-9a73-1d e3170bd3af}) failed: 8007007e xaudio2.c:1010: Test failed: xapofx1_3.dll: CreateFX({a90bc001-e897-e897-7439-43 5500000001}) failed: 80040154 xaudio2.c:1010: Test failed: xapofx1_3.dll: CreateFX({a90bc001-e897-e897-7439-43 5500000003}) failed: 80040154 xaudio2.c:1010: Test failed: xapofx1_3.dll: CreateFX({c4137916-2be1-46fd-8599-44 1536f49856}) failed: 80040154 xaudio2.c:1010: Test failed: xapofx1_3.dll: CreateFX({5039d740-f736-449a-84d3-a5 6202557b87}) failed: 80040154 xaudio2.c:1028: Test failed: xapofx1_3.dll: CreateFX({2139e6da-c341-4774-9ac3-b4 e026347f64}) failed: 8007007e xaudio2.c:1039: Test failed: xapofx1_3.dll: CreateFX({d06df0d0-8518-441e-822f-54 51d5c595b8}) failed: 8007007e xaudio2.c:1010: Test failed: xapofx1_4.dll: CreateFX({a90bc001-e897-e897-7439-43 5500000001}) failed: 80040154 xaudio2.c:1010: Test failed: xapofx1_4.dll: CreateFX({a90bc001-e897-e897-7439-43 5500000003}) failed: 80040154 xaudio2.c:1010: Test failed: xapofx1_4.dll: CreateFX({c4137916-2be1-46fd-8599-44 1536f49856}) failed: 80040154 xaudio2.c:1010: Test failed: xapofx1_4.dll: CreateFX({5039d740-f736-449a-84d3-a5 6202557b87}) failed: 80040154 xaudio2.c:1028: Test failed: xapofx1_4.dll: CreateFX({e48c5a3f-93ef-43bb-a092-2c 7ceb946f27}) failed: 8007007e xaudio2.c:1039: Test failed: xapofx1_4.dll: CreateFX({cecec95a-d894-491a-bee3-5e 106fb59f2d}) failed: 8007007e xaudio2.c:1010: Test failed: xapofx1_5.dll: CreateFX({a90bc001-e897-e897-7439-43 5500000001}) failed: 80040154 xaudio2.c:1010: Test failed: xapofx1_5.dll: CreateFX({a90bc001-e897-e897-7439-43 5500000003}) failed: 80040154 xaudio2.c:1010: Test failed: xapofx1_5.dll: CreateFX({c4137916-2be1-46fd-8599-44 1536f49856}) failed: 80040154 xaudio2.c:1010: Test failed: xapofx1_5.dll: CreateFX({5039d740-f736-449a-84d3-a5 6202557b87}) failed: 80040154 xaudio2.c:1028: Test failed: xapofx1_5.dll: CreateFX({cac1105f-619b-4d04-831a-44 e1cbf12d57}) failed: 8007007e xaudio2.c:1039: Test failed: xapofx1_5.dll: CreateFX({6a93130e-1d53-41d1-a9cf-e7 58800bb179}) failed: 8007007e xaudio2.c:1088: Tests skipped: Couldn't load xaudio2_8.dll xaudio2.c:1295: Tests skipped: XAudio 2.7 not available xaudio2.c:1320: Tests skipped: XAudio 2.8 not available 1898:xaudio2: 138 tests executed (0 marked as todo, 36 failures), 3 skipped.

MSVS OpenAL32.lib linked XACT DLLs with MSVS compiled OpenAL32.dll (renamed libopenal-1.dll)

xaudio2_7_test.exe xaudio2.c:1010: Test failed: xapofx1_1.dll: CreateFX({a90bc001-e897-e897-7439-43 5500000001}) failed: 80040154 xaudio2.c:1010: Test failed: xapofx1_1.dll: CreateFX({a90bc001-e897-e897-7439-43 5500000003}) failed: 80040154 xaudio2.c:1010: Test failed: xapofx1_1.dll: CreateFX({c4137916-2be1-46fd-8599-44 1536f49856}) failed: 80040154 xaudio2.c:1010: Test failed: xapofx1_1.dll: CreateFX({5039d740-f736-449a-84d3-a5 6202557b87}) failed: 80040154 xaudio2.c:1028: Test failed: xapofx1_1.dll: CreateFX({f5ca7b34-8055-42c0-b836-21 6129eb7e30}) failed: 8007007e xaudio2.c:1039: Test failed: xapofx1_1.dll: CreateFX({629cf0de-3ecc-41e7-9926-f7 e43eebec51}) failed: 8007007e xaudio2.c:1010: Test failed: xapofx1_2.dll: CreateFX({a90bc001-e897-e897-7439-43 5500000001}) failed: 80040154 xaudio2.c:1010: Test failed: xapofx1_2.dll: CreateFX({a90bc001-e897-e897-7439-43 5500000003}) failed: 80040154 xaudio2.c:1010: Test failed: xapofx1_2.dll: CreateFX({c4137916-2be1-46fd-8599-44 1536f49856}) failed: 80040154 xaudio2.c:1010: Test failed: xapofx1_2.dll: CreateFX({5039d740-f736-449a-84d3-a5 6202557b87}) failed: 80040154 xaudio2.c:1028: Test failed: xapofx1_2.dll: CreateFX({e180344b-ac83-4483-959e-18 a5c56a5e19}) failed: 8007007e xaudio2.c:1039: Test failed: xapofx1_2.dll: CreateFX({9cab402c-1d37-44b4-886d-fa 4f36170a4c}) failed: 8007007e xaudio2.c:1010: Test failed: xapofx1_3.dll: CreateFX({a90bc001-e897-e897-7439-43 5500000001}) failed: 80040154 xaudio2.c:1010: Test failed: xapofx1_3.dll: CreateFX({a90bc001-e897-e897-7439-43 5500000003}) failed: 80040154 xaudio2.c:1010: Test failed: xapofx1_3.dll: CreateFX({c4137916-2be1-46fd-8599-44 1536f49856}) failed: 80040154 xaudio2.c:1010: Test failed: xapofx1_3.dll: CreateFX({5039d740-f736-449a-84d3-a5 6202557b87}) failed: 80040154 xaudio2.c:1028: Test failed: xapofx1_3.dll: CreateFX({c7338b95-52b8-4542-aa79-42 eb016c8c1c}) failed: 8007007e xaudio2.c:1039: Test failed: xapofx1_3.dll: CreateFX({8bb7778b-645b-4475-9a73-1d e3170bd3af}) failed: 8007007e xaudio2.c:1010: Test failed: xapofx1_3.dll: CreateFX({a90bc001-e897-e897-7439-43 5500000001}) failed: 80040154 xaudio2.c:1010: Test failed: xapofx1_3.dll: CreateFX({a90bc001-e897-e897-7439-43 5500000003}) failed: 80040154 xaudio2.c:1010: Test failed: xapofx1_3.dll: CreateFX({c4137916-2be1-46fd-8599-44 1536f49856}) failed: 80040154 xaudio2.c:1010: Test failed: xapofx1_3.dll: CreateFX({5039d740-f736-449a-84d3-a5 6202557b87}) failed: 80040154 xaudio2.c:1028: Test failed: xapofx1_3.dll: CreateFX({2139e6da-c341-4774-9ac3-b4 e026347f64}) failed: 8007007e xaudio2.c:1039: Test failed: xapofx1_3.dll: CreateFX({d06df0d0-8518-441e-822f-54 51d5c595b8}) failed: 8007007e xaudio2.c:1010: Test failed: xapofx1_4.dll: CreateFX({a90bc001-e897-e897-7439-43 5500000001}) failed: 80040154 xaudio2.c:1010: Test failed: xapofx1_4.dll: CreateFX({a90bc001-e897-e897-7439-43 5500000003}) failed: 80040154 xaudio2.c:1010: Test failed: xapofx1_4.dll: CreateFX({c4137916-2be1-46fd-8599-44 1536f49856}) failed: 80040154 xaudio2.c:1010: Test failed: xapofx1_4.dll: CreateFX({5039d740-f736-449a-84d3-a5 6202557b87}) failed: 80040154 xaudio2.c:1028: Test failed: xapofx1_4.dll: CreateFX({e48c5a3f-93ef-43bb-a092-2c 7ceb946f27}) failed: 8007007e xaudio2.c:1039: Test failed: xapofx1_4.dll: CreateFX({cecec95a-d894-491a-bee3-5e 106fb59f2d}) failed: 8007007e xaudio2.c:1010: Test failed: xapofx1_5.dll: CreateFX({a90bc001-e897-e897-7439-43 5500000001}) failed: 80040154 xaudio2.c:1010: Test failed: xapofx1_5.dll: CreateFX({a90bc001-e897-e897-7439-43 5500000003}) failed: 80040154 xaudio2.c:1010: Test failed: xapofx1_5.dll: CreateFX({c4137916-2be1-46fd-8599-44 1536f49856}) failed: 80040154 xaudio2.c:1010: Test failed: xapofx1_5.dll: CreateFX({5039d740-f736-449a-84d3-a5 6202557b87}) failed: 80040154 xaudio2.c:1028: Test failed: xapofx1_5.dll: CreateFX({cac1105f-619b-4d04-831a-44 e1cbf12d57}) failed: 8007007e xaudio2.c:1039: Test failed: xapofx1_5.dll: CreateFX({6a93130e-1d53-41d1-a9cf-e7 58800bb179}) failed: 8007007e xaudio2.c:1096: Test failed: xaudio2_8.dll: CreateFX({a90bc001-e897-e897-7439-43 5500000001}) failed: 80040154 xaudio2.c:1096: Test failed: xaudio2_8.dll: CreateFX({a90bc001-e897-e897-7439-43 5500000003}) failed: 80040154 xaudio2.c:1096: Test failed: xaudio2_8.dll: CreateFX({c4137916-2be1-46fd-8599-44 1536f49856}) failed: 80040154 xaudio2.c:1096: Test failed: xaudio2_8.dll: CreateFX({5039d740-f736-449a-84d3-a5 6202557b87}) failed: 80040154 xaudio2.c:1115: Test failed: xaudio2_8.dll: CreateFX(CLSID_AudioVolumeMeter) fai led: 8007007e xaudio2.c:1295: Tests skipped: XAudio 2.7 not available xaudio2.c:1295: this is the last test seen before the exception 17f8:xaudio2: unhandled exception c0000005 at 71b38428

libopenal.dll.a linked XACT DLLs with cross-compiled msys-openal-1.dll (renamed libopenal-1.dll) (SAME AS ABOVE)

xaudio2_7_test.exe xaudio2.c:1010: Test failed: xapofx1_1.dll: CreateFX({a90bc001-e897-e897-7439-43 5500000001}) failed: 80040154 xaudio2.c:1010: Test failed: xapofx1_1.dll: CreateFX({a90bc001-e897-e897-7439-43 5500000003}) failed: 80040154 xaudio2.c:1010: Test failed: xapofx1_1.dll: CreateFX({c4137916-2be1-46fd-8599-44 1536f49856}) failed: 80040154 xaudio2.c:1010: Test failed: xapofx1_1.dll: CreateFX({5039d740-f736-449a-84d3-a5 6202557b87}) failed: 80040154 xaudio2.c:1028: Test failed: xapofx1_1.dll: CreateFX({f5ca7b34-8055-42c0-b836-21 6129eb7e30}) failed: 8007007e xaudio2.c:1039: Test failed: xapofx1_1.dll: CreateFX({629cf0de-3ecc-41e7-9926-f7 e43eebec51}) failed: 8007007e xaudio2.c:1010: Test failed: xapofx1_2.dll: CreateFX({a90bc001-e897-e897-7439-43 5500000001}) failed: 80040154 xaudio2.c:1010: Test failed: xapofx1_2.dll: CreateFX({a90bc001-e897-e897-7439-43 5500000003}) failed: 80040154 xaudio2.c:1010: Test failed: xapofx1_2.dll: CreateFX({c4137916-2be1-46fd-8599-44 1536f49856}) failed: 80040154 xaudio2.c:1010: Test failed: xapofx1_2.dll: CreateFX({5039d740-f736-449a-84d3-a5 6202557b87}) failed: 80040154 xaudio2.c:1028: Test failed: xapofx1_2.dll: CreateFX({e180344b-ac83-4483-959e-18 a5c56a5e19}) failed: 8007007e xaudio2.c:1039: Test failed: xapofx1_2.dll: CreateFX({9cab402c-1d37-44b4-886d-fa 4f36170a4c}) failed: 8007007e xaudio2.c:1010: Test failed: xapofx1_3.dll: CreateFX({a90bc001-e897-e897-7439-43 5500000001}) failed: 80040154 xaudio2.c:1010: Test failed: xapofx1_3.dll: CreateFX({a90bc001-e897-e897-7439-43 5500000003}) failed: 80040154 xaudio2.c:1010: Test failed: xapofx1_3.dll: CreateFX({c4137916-2be1-46fd-8599-44 1536f49856}) failed: 80040154 xaudio2.c:1010: Test failed: xapofx1_3.dll: CreateFX({5039d740-f736-449a-84d3-a5 6202557b87}) failed: 80040154 xaudio2.c:1028: Test failed: xapofx1_3.dll: CreateFX({c7338b95-52b8-4542-aa79-42 eb016c8c1c}) failed: 8007007e xaudio2.c:1039: Test failed: xapofx1_3.dll: CreateFX({8bb7778b-645b-4475-9a73-1d e3170bd3af}) failed: 8007007e xaudio2.c:1010: Test failed: xapofx1_3.dll: CreateFX({a90bc001-e897-e897-7439-43 5500000001}) failed: 80040154 xaudio2.c:1010: Test failed: xapofx1_3.dll: CreateFX({a90bc001-e897-e897-7439-43 5500000003}) failed: 80040154 xaudio2.c:1010: Test failed: xapofx1_3.dll: CreateFX({c4137916-2be1-46fd-8599-44 1536f49856}) failed: 80040154 xaudio2.c:1010: Test failed: xapofx1_3.dll: CreateFX({5039d740-f736-449a-84d3-a5 6202557b87}) failed: 80040154 xaudio2.c:1028: Test failed: xapofx1_3.dll: CreateFX({2139e6da-c341-4774-9ac3-b4 e026347f64}) failed: 8007007e xaudio2.c:1039: Test failed: xapofx1_3.dll: CreateFX({d06df0d0-8518-441e-822f-54 51d5c595b8}) failed: 8007007e xaudio2.c:1010: Test failed: xapofx1_4.dll: CreateFX({a90bc001-e897-e897-7439-43 5500000001}) failed: 80040154 xaudio2.c:1010: Test failed: xapofx1_4.dll: CreateFX({a90bc001-e897-e897-7439-43 5500000003}) failed: 80040154 xaudio2.c:1010: Test failed: xapofx1_4.dll: CreateFX({c4137916-2be1-46fd-8599-44 1536f49856}) failed: 80040154 xaudio2.c:1010: Test failed: xapofx1_4.dll: CreateFX({5039d740-f736-449a-84d3-a5 6202557b87}) failed: 80040154 xaudio2.c:1028: Test failed: xapofx1_4.dll: CreateFX({e48c5a3f-93ef-43bb-a092-2c 7ceb946f27}) failed: 8007007e xaudio2.c:1039: Test failed: xapofx1_4.dll: CreateFX({cecec95a-d894-491a-bee3-5e 106fb59f2d}) failed: 8007007e xaudio2.c:1010: Test failed: xapofx1_5.dll: CreateFX({a90bc001-e897-e897-7439-43 5500000001}) failed: 80040154 xaudio2.c:1010: Test failed: xapofx1_5.dll: CreateFX({a90bc001-e897-e897-7439-43 5500000003}) failed: 80040154 xaudio2.c:1010: Test failed: xapofx1_5.dll: CreateFX({c4137916-2be1-46fd-8599-44 1536f49856}) failed: 80040154 xaudio2.c:1010: Test failed: xapofx1_5.dll: CreateFX({5039d740-f736-449a-84d3-a5 6202557b87}) failed: 80040154 xaudio2.c:1028: Test failed: xapofx1_5.dll: CreateFX({cac1105f-619b-4d04-831a-44 e1cbf12d57}) failed: 8007007e xaudio2.c:1039: Test failed: xapofx1_5.dll: CreateFX({6a93130e-1d53-41d1-a9cf-e7 58800bb179}) failed: 8007007e xaudio2.c:1096: Test failed: xaudio2_8.dll: CreateFX({a90bc001-e897-e897-7439-43 5500000001}) failed: 80040154 xaudio2.c:1096: Test failed: xaudio2_8.dll: CreateFX({a90bc001-e897-e897-7439-43 5500000003}) failed: 80040154 xaudio2.c:1096: Test failed: xaudio2_8.dll: CreateFX({c4137916-2be1-46fd-8599-44 1536f49856}) failed: 80040154 xaudio2.c:1096: Test failed: xaudio2_8.dll: CreateFX({5039d740-f736-449a-84d3-a5 6202557b87}) failed: 80040154 xaudio2.c:1115: Test failed: xaudio2_8.dll: CreateFX(CLSID_AudioVolumeMeter) fai led: 8007007e xaudio2.c:1295: Tests skipped: XAudio 2.7 not available xaudio2.c:1295: this is the last test seen before the exception 0220:xaudio2: unhandled exception c0000005 at 71b38428

I have a new theory and some results.

Previously the Wine-4.2 XAudio2 files built with the library file specified with the variable ac_cv_lib_soname_openal would always define the dependency of libopenal-1.dll as named with the default files installed with pacman -S mingw-w64-i686-openal within MSYS2.

After successfully cross-compiling OpenAL in MSYS2 using i686-w64-mingw32 and creating a massive 28MB msys-openal-1.dll, I was also left with a new version of libopenal.dll.a derived from that build. I can compile wine-4.2 with this new libopenal.dll.a and the dependency filename for the Wine-4.2 XAudio2 files changes to msys-openal-1.dll. This convinces me that I am selecting the correct file to link the Wine-4.2 XAudio2 files to OpenAL Soft.

All the XAudio2 DLLs paired with the cross-compiled msys-openal-1.dll and tested with xaudio2_7_tests.exe continue to only spit out an ACCESS VIOLATION. I have used another tool DEPENDENCY WALKER to cross-reference with DependencyGUI. The new files are available here:

wine-4.2-xact-msys-openal-1.dll-linked-bin+src.zip

msys-openal-1-soft-1.23.1+jack-bin+src.zip

I've had the privilege to talk with Andrew Eikum, creator of the code for the XAudio2 reimplementation based on the OpenAL Soft.

On Mon, May 13, 2024, at 4:57 PM, Alex Schroedsen wrote:

My name is Alex Schroedsen. I've recently been working on getting older games to support spatial audio on Windows through OpenAL Soft. I see that you've worked on a project some time back that achieved xaudio2 compatibility on top of OpenAL Soft. My mission is to get XAudio2 games working on top of OpenAL Soft within Windows with the code you created. I am wondering if you'd be willing/interested to help with this mission?

Hi Alex!

Interesting to see this old code being brought back to life! Hahah. I figured it was completely obsolete when we switched over to FAudio.

I have to admit I'm not 100% sure what your goal is here. Wine's XAudio2 and FAudio should work on Windows, I believe with support for spatial audio & surround sound. I think the project is fairly feature-complete at this point. Is there something you get from OpenAL that FAudio can't provide?

You can see the shenanigans I've already been up to here:

Work in progress: https://github.com/kcat/openal-soft/issues/997

For debugging these failures, are you familiar with the WINEDEBUG environment variable for getting Wine trace logs? That will hopefully give you more information about what is failing.

Andrew

I asked a few questions and he got back to me with useful info.

On Tue, May 14, 2024, at 11:19 AM, Alex Schroedsen wrote:

The goal here is that with OpenAL Soft outputting third order ambisonics, connected to a VST host and then to windows using ASIO and JACK connections, there is the possibility of users to use the GUI/user friendly ALLRAD Decoder from IEM Suite to create their own speaker configurations while also allowing the appealing ideas of bit-perfect output and low latency. No mathematical or command line experience necessary.

The reasons that I currently believe in bringing your code back to life rather than using FAudio with SDL are:

[Ambisonic support, channel count, low latency, bit perfect and ease of use] SDL outputting to the WASAPI backend within Windows seems to be limited to a 7.1 surround (8 channel) output whereas OpenAL Soft can output 3rd order ambisonics (16 channels) and be decoded into any speaker configuration that can be created within IEM’s ALLRAD Decoder. With the added channel counts Made possible with JACK and ASIO, any speaker configuration within and maybe outside of reason should be possible.

[Overcoming proprietary spatial audio limitation]Windows Spatial APIs seem to currently be limited to expensive proprietary codecs like dolby atmos. Making custom speaker configurations possible with discrete channels will be a major victory for those like myself who are exploring spatial audio in gaming.

All right. I would still suggest avoiding the old code. It's definitely buggy and doesn't have the benefit of 5+ years of work on FAudio and Wine's integration with it.

Instead, I think you could try writing an OpenAL backend for FAudio, like it already has for SDL and WASAPI. I'd look at how those FAudio_platform* files interact with the public XAudio2 interface, and then implement a platform that targets OpenAL. Make sure it passes FAudio's and Wine's tests. If needed, you could then extend the connection between the internal platform API and the XAudio2 API, to meet your spatial audio needs (e.g. pass positional data from the XAudio2 interface through to OpenAL, or whatever is you'd like to achieve with it).

With these benefits in mind I have a few questions.

I have been using DependenciesGUI (https://lucasg.github.io/Dependencies/) to probe for what program is using what DLLs. Upon trying different combinations I am able to debug some programs in Microsoft Visual Studio to see the load order of Windows DLLs and there are MANY more DLLs that a program actually uses beyond what DepenencyGUI lists. For instance, I've been testing the crosscompilation builds with Sonic Mania for Windows. It doesn't list xaudio2_7.dll as a dependency within DependenciesGUI but when it crashes and I debug it within MSVS, it lists the xaudio2_7.dll and also winmm.dll (and many more) as DLLs that are loaded.

Yes, it looks like that program only shows static dependencies, not libraries loaded at runtime. COM stuff, like XAudio2, is loaded at runtime so won't show up as a static dependency.

Crosscompiled DLLs from Wine-4.2 for the xaudio2_7.dll list only libwine.dll and libopenal-1.dll as their non-windows-native dependency. What files might the wine-4.2 implementation of xaudio2_7.dll depend on? (Other than libwine.dll and libopenal-1.dll)

I know it's something people do, but I've actually almost never used Wine libraries on Windows. So I don't know the details of how to do that. Hopefully there are Wine docs that can help you there, or maybe you can ask for help from someone else.

In order to support DirectSound3D games, kcat has developed a translation layer to convert DirectSound3D calls to OpenAL calls. This layer is called DSOAL (DirectSound over OpenAL). DSOAL requests that OpenAL Soft open a specific stringified WASAPI device GUID for it's output device. In order to make DSOAL happy when OpenAL Soft is opening a JACK backend instead of WASAPI, I was taught how to create a custom device name that matches the stringified WASAPI GUID that DSOAL expects OpenAL to select. https://github.com/kcat/openal-soft/issues/984#issuecomment-2060767239 I am wondering if the same situation might apply to the wine-4.2 implementation of xaudio2_7.dll. If I were to request a custom name for the JACK device, is there a naming convention that might bring me success?

Some games are picky about device names (especially if they're trying to use several audio APIs at once and correlate the devices across them), but it's fairly uncommon for XAudio2 clients. Mostly they just open the default device. I think you're on the right path with running the Wine test suite against your implementation; if there are requirements there, then there should be tests to validate them.

When you were debugging this code back in the day: Was there an application that you used to test its functionality other than xaudio2_7_tests.exe? I'm guessing Skyrim was a big one. I am interested in the name of something that you consider to be a model example of how well the wine-4.2 implementation of xaudio2 can work.

Oohhh unfortunately I've long forgotten. Skyrim was definitely one, DOOM 2016 was another. You could maybe dig through the Wine bug tracker and bug URLs from the Git log around that era.

Is it okay if I copy our messages over to the github issue so that others can benefit from your knowledge as well?

Sure. Happy to help.

I have not known about the WINEDEBUG environment variable. I will learn about it and see if I can answer some of my own questions. If you happen to know the answers to my questions off the top of your head, your help will always be appreciated.

Yeah, WINEDEBUG is how you get the output from all those TRACEs and WARNs and ERRs you see in the code. I wrote a blog post about it you might find helpful (be warned, some of this series is definitely out of date by now): https://www.codeweavers.com/blog/aeikum/2019/1/15/working-on-wine-part-4-debugging-wine

Andrew

Thus I am currently working on learning to use the WINE debugging environment variable. I do not know how to code in c, c++ or much ore about these audio APIs so I'm continuing with my goal to compile the old code. I am also working on learning how to use GDB and GDBserver in order to debug xaudio2_7.dll using xaudio2_7_tests.exe. I will report back with news as I have it.

Honestly I'd have considered starting a bit more fresh, ala DSOAL. Start by building a native Windows DLL for xaudio2, exporting the functions it's supposed to export (empty to start with, just do nothing and return error), then link to and use the OpenAL Soft Windows DLL to start implementing it (using the old Wine xaudio2 code as a basis). There shouldn't be a need to cross-compile anything if you're on Windows, and only cross-compile the new XAudio2 dll and OpenAL Soft to Windows on other systems. You shouldn't need Wine if you're on Windows (and on Linux using Wine, it can use normal Windows DLLs, so the build process doesn't need Wine).

I have made more progress and had new discoveries. I would start from scratch on this, but I am currently so novice that I am doing what I can, which is looking at references in the current code and seeing what does not match up. I am able to learn by fiddling with an existing implementation and seeing how its pieces interact whereas learning three languages (C, XAudio2, OpenAL) at once and needing to correctly implement them all simultaneously is a far more ominous task.

Yesterday I dived into creating a wrapper that would hopefully pull the name of the device that audio2_7.dll would request OpenAL Soft to open. Following the instructions here API Interception via DLL Redirection. I successfully created a stub that would pass on all communications to the real OpenAL32.dll. I then fiddled with trying to convert the device name called with alcOpenDevice to a string and put up a message box with the device name.

However I had a realization as I sifted through the code of xaudio_dll.c that it never calls the alcOpenDevice function. It instead calls the alcLoopbackOpenDeviceSOFT function. Sidenote-I also noticed that the function is specified to open the default OpenAL Soft device by specifying NULL in the device name parameter. So the original device naming convention problem is off the table. I looked up alcLoopbackOpenDeviceSOFT to learn it’s similarities and differences to the classic alcOpenDevice and found this article openalext/ALC_SOFT_loopback.txt at master · openalext/openalext.

Issues

Q: How are loopback devices created and handled?
A: Loopback devices are created using the alcLoopbackOpenDeviceSOFT
   function. They are closed using the standard alcCloseDevice function.
   Note that since alcLoopbackOpenDeviceSOFT is a driver-specific export
   and returns a driver-specific device handle, the ALC and AL functions
   used with such a device must come from the same driver.

Q: How is the device's render format specified?
A: By passing ALC_FORMAT_CHANNELS_SOFT, ALC_FORMAT_TYPE_SOFT, and
   ALC_FREQUENCY attributes when creating a context. If more than one
   context is created, the device will use the last format specified. It
   is also considered an error to create a context without specifying all
   three attributes.

Q: Does the device update automatically over time?
A: No. The speed of rendering is controlled entirely by the app, so the
   faster the app calls to render audio, the faster the device's (and
   subsequently, source's and buffer's) state progresses. This allows an
   app to, for example, render an audio mix to encode with a video without
   having to wait the amount of time it would take to actually play.

Q: Are all context attributes supported with loopback devices?
A: All standard ones except for ALC_SYNC and ALC_REFRESH. The precise
   meaning of ALC_SYNC is not well defined for this situation, and the
   refresh rate is controlled by the app to be as fast or slow as it
   Wants.

So far I see a few challenges to be tackled in order to make this implementation work. In order to modify it to support the alcOpenDevice rather than alcLoopbackOpenDeviceSOFT.

The xaudio2_dll.c will need to first call the original alcOpenDevice. Now we have the challenge of setting automatic updates to occur for rendering to the non-loopback output device.
I’m assuming that the original device interacts with ALSA as it is built for linux. Or it may interact with an emulated mmdevapi. Detangling the separate drivers of JACK vs WASAPI/DirectSound/winMM will be another thing.
Channel masks only go up to 7POINT1 in xaudio_dll.c. I'm guessing I’d also have to figure out how to spoof a supported channel mask to keep xaudio_dll.c happy when OpenAL Soft tries to output 3rd order ambisonics.
We may also need to remove the dependency on another function alcRenderSamplesSOFT, which seems like it might be a clocking mechanism for alcLoopbackOpenDeviceSOFT.

At this point I am wondering if all of this work will amount to OpenAL Soft not actually rendering positional data for height speakers, but only acting as an upmixer for the same set of channel masks that XAudio2 supports in this implementation. If you have an idea of how to solve these issues and get this implementation working, any help is appreciated. I remain interested in seeing this through to the point where we can at least see this existing implementation work and then perhaps be inspired to explore further, creating an improved implementation with the new and more feature complete FAudio or building something completely new.

Here is the file I've been looking through xaudio_dll.c.txt And the project files xaudio2_7.zip

The xaudio2_dll.c will need to first call the original alcOpenDevice. Now we have the challenge of setting automatic updates to occur for rendering to the non-loopback output device.

The biggest problem will be the processing callbacks. I don't know if OpenAL Soft's callback buffers will be enough to handle IXAudio2VoiceCallback_OnVoiceProcessingPassStart/End and related, or if it would even be safe to. Normal playback devices don't have a way to handle IXAudio2EngineCallback_OnProcessingPassStart/End either. That's what the loopback device is for, to allow invoking the callbacks more properly before and after each render/mixing update (which is what alcRenderSamplesSOFT does). Using the loopback device, though, then requires having some other method to output the mixed samples; in the case of that code, it uses WASAPI.

Channel masks only go up to 7POINT1 in xaudio_dll.c. I'm guessing I’d also have to figure out how to spoof a supported channel mask to keep xaudio_dll.c happy when OpenAL Soft tries to output 3rd order ambisonics.

Depending on what XAudio allows apps to do with the audio, it may be as simple as ignoring the device's channel mask (store it and report it back if queried, but set whatever output you want with OpenAL Soft regardless). If the app expects to have access to or control of the individual output channels, though, it can get more complicated depending on what exactly it does.

Thank you so much for explaining why the alcLoopbackOpenDeviceSOFT and alcRenderSamplesSOFT functions are used in this case. So -correct me if I'm wrong- xaudio_dll.c makes a request to OpenAL Soft to render audio data, OpenAL Soft hands the rendered audio data back to xaudio_dll.c and xaudio_dll.c then interacts with a WASAPI audio endpoint to output the rendered audio. The problem with this setup is that with every single chunk of audio data that is rendered, there is a start and stop by the API, which would usually not exist in practically all other audio backends (than WASAPI). Those other backends, I am guessing, specify channel mask, word format, frequency, subsequently the tedious callbacks for buffering would normally be abstracted behind the backend library and driver where applicable.

Assuming the above is correct, looking at the channel mask code in xaudio_dll.c

static DWORD get_channel_mask(unsigned int channels)
{
    switch(channels){
    case 0:
        return 0;
    case 1:
        return KSAUDIO_SPEAKER_MONO;
    case 2:
        return KSAUDIO_SPEAKER_STEREO;
    case 3:
        return KSAUDIO_SPEAKER_STEREO | SPEAKER_LOW_FREQUENCY;
    case 4:
        return KSAUDIO_SPEAKER_QUAD;    /* not _SURROUND */
    case 5:
        return KSAUDIO_SPEAKER_QUAD | SPEAKER_LOW_FREQUENCY;
    case 6:
        return KSAUDIO_SPEAKER_5POINT1; /* not 5POINT1_SURROUND */
    case 7:
        return KSAUDIO_SPEAKER_5POINT1 | SPEAKER_BACK_CENTER;
    case 8:
        return KSAUDIO_SPEAKER_7POINT1_SURROUND; /* Vista deprecates 7POINT1 */
    }
    FIXME("Unknown speaker configuration: %u\n", channels);
    return 0;
}

The above speaker configurations may be able to be augmented according to this KSAUDIO_CHANNEL_CONFIG structure (ksmedia.h) there is also the KSAUDIO_SPEAKER_DIRECTOUT configuration. Perhaps we could avoid messing with the way that xaudio_dll.c and OpenAL Soft interact and instead just widen the pipe through xaudio_dll.c. Something like this (and other associated parts of the code):

static DWORD get_channel_mask(unsigned int channels)
{
    switch(channels){
    case 0:
        return 0;
    case 1:
        return KSAUDIO_SPEAKER_MONO;
    case 2:
        return KSAUDIO_SPEAKER_STEREO;
    case 3:
        return KSAUDIO_SPEAKER_STEREO | SPEAKER_LOW_FREQUENCY;
    case 4:
        return KSAUDIO_SPEAKER_QUAD;    /* not _SURROUND */
    case 5:
        return KSAUDIO_SPEAKER_QUAD | SPEAKER_LOW_FREQUENCY;
    case 6:
        return KSAUDIO_SPEAKER_5POINT1; /* not 5POINT1_SURROUND */
    case 7:
        return KSAUDIO_SPEAKER_5POINT1 | SPEAKER_BACK_CENTER;
    case 8:
        return KSAUDIO_SPEAKER_7POINT1_SURROUND; /* Vista deprecates 7POINT1 */
    case 9:
        return KSAUDIO_SPEAKER_DIRECTOUT; /* Second Order Ambisonics */
    case 16:
        return KSAUDIO_SPEAKER_DIRECTOUT; /* Third Order Ambisonics */
    }
    FIXME("Unknown speaker configuration: %u\n", channels);
    return 0;
}

Now if this is using WASAPI specifically, we might run into that 8 channel issue again, but if it can be forced to use Kernel Streaming instead, we can likely get away with it. If I had a convenient way to loopback 16 channels of audio using Kernel Streaming, I’d just do that and use ASIO4ALL to bridge it into a VST host, but I don’t know of a free solution at this time. Thin Audio Gateway seems like a potential solution, unfortunately it is closed source and proprietary.

Hijack the DLL for emulating the windows audio endpoint for WINE (mmdevapi.dll?) and redirect the output of that into JACK instead of ALSA.

-OR-

Find a way to rewrite the WASAPI output in xaudio_dll.c to output audio data into JACK instead of WASAPI.

Thinking about this a little more, the client application accessing xaudio_dll.c probably wouldn’t like seeing (or be able to request) an undefined speaker config… Perhaps the returning audio data can be classified with a different set of handles/variables or something to keep the client application happy.

Given my guesses turn out to be correct. Do either of these sound like a better way to avoid dealing with the callbacks? Or perhaps they just move the work of dealing with the unique callback behavior to another area… Thank you again for your time and effort in guiding and teaching me on this journey.

Today I dug more into how xaudio_dll.c sends audio to WASAPI after receiving it back from OpenAL Soft. I am currently brainstorming how to replace the WASAPI backend in xaudio_dll.c with JACK. I have noticed a few things that I am interested in clearing up. On lines 1792 to 1821 xaudio_dll.c tries to open a context with OpenAL Soft and specific channel configurations are referenced in the code:

/* setup openal context */
attrs[0] = ALC_FORMAT_CHANNELS_SOFT;
switch(inputChannels){
case 1:
    attrs[1] = ALC_MONO_SOFT;
    break;
case 2:
    attrs[1] = ALC_STEREO_SOFT;
    break;
case 4:
    attrs[1] = ALC_QUAD_SOFT;
    break;
case 6:
    attrs[1] = ALC_5POINT1_SOFT;
    break;
case 7:
    attrs[1] = ALC_6POINT1_SOFT;
    break;
case 8:
    attrs[1] = ALC_7POINT1_SOFT;
    break;
default:
    WARN("OpenAL doesn't support %u channels\n", inputChannels);
    LeaveCriticalSection(&This->lock);
    return AUDCLNT_E_UNSUPPORTED_FORMAT;
}

So now we have a set of WASAPI/KERNEL STREAMING channel masks and the above designations for OpenAL Soft. Searching for ALC_7POINT1_SOFT in the project files of OpenAL Soft 1.23.1, I found references to it in alc.cpp, alext.h and alloopback.c. Looking in alc.cpp, I found some references to the channel masks in lines 1496 through 1512 and a couple others elsewhere.

ALCenum EnumFromDevFmt(DevFmtChannels channels)
{
    switch(channels)
    {
    case DevFmtMono: return ALC_MONO_SOFT;
    case DevFmtStereo: return ALC_STEREO_SOFT;
    case DevFmtQuad: return ALC_QUAD_SOFT;
    case DevFmtX51: return ALC_5POINT1_SOFT;
    case DevFmtX61: return ALC_6POINT1_SOFT;
    case DevFmtX71: return ALC_7POINT1_SOFT;
    case DevFmtAmbi3D: return ALC_BFORMAT3D_SOFT;
    /* FIXME: Shouldn't happen. */
    case DevFmtX714:
    case DevFmtX3D71: break;
    }
    throw std::runtime_error{"Invalid DevFmtChannels: "+std::to_string(int(channels))};
}

I’m guessing that to handle the 16 channel signal of a third order ambisonic signal, ALC_BFORMAT3D_SOFT would need to be added as a supported format in the xaudio_dll.c file. Additional to this, however, I am guessing that an app would also need to specifically ask for ambisonic 3D through the XAudio API, which is not implemented. My first question is: when channels=ambi3 is set in %appdata%\alsoft.ini; does this override the channel mask being rendered to a loopback interface? Does an OpenAL Soft loopback interface support the ALC_BFORMAT3D_SOFT channel format?

Towards the goal of modifying xaudio_dll.c rather than OpenaL Soft. It seems to me that in order to make the channel mask format requested by XAudio and the channel mask format output/returned from OpenAL Soft loopback to coexist, functionality needs to be implemented to emulate support (return whatever the XAudio app is asking for, like you said) on the This->fmt.dwChannelMask, pChannelMask and This->fmt.Format.nChannels designators and a separate set of designators created to specify the output channel count negotiated with JACK (ignoring channel mask). Somehow OpenAL Soft would need to share the current channel count specified in channels=ambi3 within %appdata%\alsoft.ini with xaudio_dll.c. (references to the channel mask storing variables from xaudio_dll.c seen below)

memcpy(&This->fmt, fmt, sizeof(WAVEFORMATEX) + fmt->cbSize);
This->fmt.Format.nChannels = inputChannels;
This->fmt.Format.nSamplesPerSec = inputSampleRate;
This->fmt.Format.nBlockAlign = This->fmt.Format.nChannels * This->fmt.Format.wBitsPerSample / 8;
This->fmt.Format.nAvgBytesPerSec = This->fmt.Format.nSamplesPerSec * This->fmt.Format.nBlockAlign;
This->fmt.dwChannelMask = get_channel_mask(This->fmt.Format.nChannels);

I am still working on learning what the function of these IXAudio2EngineCallback_OnProcessingPassStart/End callbacks are and why they're going to be a challenge to deal with. I have been looking at JACK-AUDIO-CONNECTION-KIT: Setting Client Callbacks and asking carefully formulated questions to Meta AI to try to learn a bit more about how the WASAPI backend code works and on the sequence of transactions between xaudio_dll.c, OpenAL Soft and WASAPI. Meta AI suggested using jack_set_process_callback (jack_client_t client, JackProcessCallback process_callback, void arg) to interface with XAudio’s callbacks. I read the Microsoft page for the IXAudio2EngineCallback_OnProcessingPassStart/End callbacks and they seem like they're for buffer underrun reporting and compensation. Can you explain a bit more about when they happen, why they happen and what about OpenAL Soft would make working with them difficult? I'm also curious to know if you think the JACK jack_set_process_callback has any value in this situation?

For creating the OpenAL context, you basically need to ignore inputChannels and set the attributes to render 3rd order ambisonics. ALC_SOFT_loopback_bformat.

attr[0] = ALC_FORMAT_CHANNELS_SOFT;
attr[1] = ALC_BFORMAT3D_SOFT;
attr[2] = ALC_AMBISONIC_LAYOUT_SOFT;
attr[3] = ALC_ACN_SOFT; // or ALC_FUMA_SOFT
attr[4] = ALC_AMBISONIC_SCALING_SOFT;
attr[5] = ALC_SN3D_SOFT; // or ALC_N3D_SOFT or ALC_FUMA_SOFT
attr[6] = ALC_AMBISONIC_ORDER_SOFT;
attr[7] = 3;
...

(make sure to update the indices for the remaining attributes, and the size of the attr array). You can't just make that change, though, since the WASAPI device is expecting a normal channel configuration and the buffer won't be large enough. It would need to be done at the same time as moving to JACK.

Interfacing with JACK is going to be pretty tricky. The problem is that JACK's process callback runs in a real-time thread, which isn't allowed to do things that could block or waste too much time. Since XAudio allows apps to set their own callbacks to run when processing, and do_engine_tick does make non-real-time-safe calls, it needs to be done outside of the JACK callback. You'd essentially need to manually allocate an appropriate sized buffer that do_engine_tick can write into, and the JACK callback will read from (using an atomic boolean to synchronize).

the WASAPI backend within Windows seems to be limited to a 7.1 surround (8 channel) output

Not really

whereas OpenAL Soft can output 3rd order ambisonics (16 channels) and be decoded into any speaker configuration

It's not like openal-soft ain't still using WASAPI under the hood?

Windows Spatial APIs seem to currently be limited to expensive proprietary codecs

It might, but so then? Shouldn't your effort quite more focused on first figuring this one out? For starters, what's AUDCLNT_STREAMOPTIONS_AMBISONICS?

Instead, I think you could try writing an OpenAL backend for FAudio

Or even better perhaps, why aren't just trying to repurpose FAudio, period? Like.. I can see the merits of not replicating code, but I feel importing OAL's hrtf code in there would be somehow less hacky than stacking audio layers on top of audio layers.

And why aren't you starting by resurrecting the late x3daudio1_7_hrtf project btw?

kcat / openal-soft

Support for XAudio2 via the Wine-4.2 compatibility layer #997