Add optional support for WMA via FFmpeg

[flibitijibibo]

The likelihood of us ever getting a permissively-licensed WMA/xWMA decoder is close to zero, but that should only affect commercial FNA projects that have access to the source data (and thus can choose a different compression method). For projects like the COM wrapper, Wine, and @0x0ade's XnaToFna, we can use something like FFmpeg to deal with this.

@aeikum has already gotten started on this; adding the decoder is simple enough but we'll also have to fuss with the decoders a little bit due to the various ways WMA sources get special treatment in the spec:

https://github.com/aeikum/FAudio/commit/2a90621e071e9a547ba54462bdc87f3c56529618

https://github.com/aeikum/FAudio/commit/dd1902945e269c96c81480e98155a6af4de506e5

[flibitijibibo]

For anyone doing this with XACT as their motivation, I sat down and figured out how to get the seek table data from the WaveBanks:

https://github.com/FNA-XNA/FAudio/commit/ccd18c84e640e9a9925a2eec758e6996cc87d9b2

These tables can get sent to SubmitSourceBuffer via the BUFFER_WMA parameter, and should be the equivalent of the dpds table that you would get from a non-XACT WMA file.

[JohanSmet]

I've been looking into this a bit. I started from Andrew's work and finished integrating it with FAudio.

ATM it's successfully playing a xWMA file (tested with 1 and 2 channels). It's in the ffmpeg branch of my repo. It needs more polish and definitely more testing :-) Right now I've only tested with a small utility program I wrote myself ...

I'll rebase against the current master later tonight and try to run some more tests (maybe together with the COM wrapper).

[flibitijibibo]

Neat! Another possible test is facttool with WMA WaveBanks. Samples include the Windows versions of Owlboy, Murder Miners, Skulls of the Shogun (music), Apotheon (music), and Cryptark (music, voice).

The one catch is that I haven't quite finished the WMA submit part yet... right now the walls are here:

https://github.com/FNA-XNA/FAudio/blob/master/src/FACT.c#L1443 https://github.com/FNA-XNA/FAudio/blob/master/src/FACT_internal.c#L1598

[GloriousEggroll]

@JohanSmet just out of curiosity I tried to compile your branch to test some games that I know have wma issues and got this error:

/usr/lib/gcc/x86_64-w64-mingw32/8.2.0/../../../../x86_64-w64-mingw32/bin/ld: src/FAudio.o: in function `FAudio_CreateSourceVoice':
/home/ge/FACT/src/FAudio.c:260: undefined reference to `FAudio_FFMPEG_init'
collect2: error: ld returned 1 exit status
make: *** [Makefile:79: all] Error 1

used the following method:

git clone git://github.com/JohanSmet/FACT.git cd FACT git fetch git checkout ffmpeg source cpp/scripts/cross_compile_64 make cd cpp make

also worth noting @flibitijibibo for Arch users MINGWROOT=sys-root/mingw is not needed in the paths in the cross_compile_scripts as the bin folder resides directly in the MINGW=x86_64-w64-mingw32 path (when using mingw-w64-sdl2 from the aur). As this is probably specific to Arch, it probably doesn't need to be changed. just making a note in case anyone else is attempting this.

"btw I use Arch."

[flibitijibibo]

I think you need to set FAUDIO_FFMPEG=1 when building with ffmpeg support; that error will need to be fixed for non-ffmpeg builds.

EDIT: Also something else that needs to be looked at is mingw32-make support; normally you would use this instead of a special script but for whatever reason this generates a broken FAudio DLL for me, and I'm not sure why yet. If we can fix support for that we get really close to not needing a script at all (except maybe to set variables for COM and ffmpeg support).

[JohanSmet]

Ah, I forgot an #ifdef test in FAudio.c, that's fixed now. Thanks for the heads-up, @GloriousEggroll !

Yes, @flibitijibibo is right. You'll need to export FAUDIO_FFMPEG=1 to enable FFmpeg support. And you'll also need a FFmpeg install (for the platform you're cross-compiling to) in ../ffmpeg. I haven't tested this with cross-compiling yet, don't know if this will compile cleanly. The build system changes for this patch need some more attention. It would probably be nice to link against the system's FFmpeg for native linux builds, etc...

Please note that playing partial buffers (PlayBegin/PlayLength != 0) is probably broken at the moment. Games that use this feature might sound a bit weird for now :-)

[flibitijibibo]

Something that is not entirely apparent in the most recent documentation is that there are a couple rules for xWMA that should make life easier for us. The huge thing is that subregion looping is not supported at all, so for that format we can assert(LoopBegin == 0 && (LoopLength == 0 || LoopLength == wholeFileSize)) for any LoopCount > 0. The trouble is solely with PlayBegin/PlayLength, which does not appear to have any explicit rules but the documentation vaguely implies that there are probably alignment rules similar to ADPCM.

[JohanSmet]

The docs for SubmitSourceBuffer say that for XMA buffers the play regions must begin or end on a 128 sample boundary in the decoded audio. I don't see how this helps as it's my understanding that the number of decoded samples from a WMA packet can vary between packets.

Right now I save the FAudioBufferWMA data in SubmitSourceBuffer and use that in the decoder to quickly seek to the wanted position. As a side effect subregion looping just works because that's handled at a higher layer in FAudio_INTERNAL_DecodeBuffers.

I tested Murder Miners and Skulls of the Shogun a bit and these seem to work fine. For these tests I only replaced the xaudio2 dll's not xactengine.

[flibitijibibo]

Cool, if the XAudio2 side works then the XACT side shouldn't be too hard after that. I'm thinking the alignment relates to the packet sizes but if the block sizes can be arbitrary then maybe I'm overthinking this...

The one thing I would trace is Shogun's SubmitSourceBuffer calls, as the music WaveBank in particular should be streaming the data instead of loading it all at once; how they submit it should give us an idea of how FACT should be doing it, and that'll keep us from accidentally coming up with a new use case.

A couple more WMA samples if you want them: Shenmue I & II, Dead Rising 2, Castle Crashers, and of course, Skyrim... please let this work for Skyrim...

[GloriousEggroll]

A fun test that might be good for FAudio testing/compatibility is Warframe, as it requires the following patches from wine-staging:

xaudio2_7-CreateFX-FXEcho
xaudio2_7-OnVoiceProcessingPassStart
xaudio2_7-WMA_support
xaudio2_CommitChanges

-EDIT- I've made a modified branch for the launcher for warframe standalone that does not include directx or any audio registry entries:
packages needed: winetricks, tar, lzma/unlzma, curl, wget, md5sum

https://gitlab.com/GloriousEggroll/warframe-linux/tree/no-directx
Just run ./install.sh then run warframe in terminal

the custom launcher is needed due to a bootloop issue with the official launcher.

[aeikum]

Just so everyone's aware, Wine does have a small xaudio2 test suite. As you start to understand the behavior of real games, you can start to write tests there and verify the behavior of Microsoft's xaudio2. We also have a testbot infrastructure you can use to run tests on Windows. The test loop will go faster if you have a Windows VM, but if you don't, you can use the testbot. https://testbot.winehq.org/

[JohanSmet]

Ok, I tested Skyrim (not SE) on Windows 8.1. There was an issue with submix voices but other than that it sounds fine to me. But I did only play for about a minute (and still managed to die, don't seem to remember how to play ...)

Skyrim uses processing stage 0xFFFFFFFF (-1) when creating submix voices. This caused FAudio_INTERNAL_UpdateEngine to stall when iterating over all the processing stages between 0 and 2^32-1 and playing no audio. I fixed this by insertion sorting the submix voices on their processing stage at creation time and just iterating over that list when mixing (see: 7fb48736879abb939a5eb3c6d1b29bee6ece75c5).

Tomorrow I'll clean up the rough edges (I think I broke the non-ffmpeg build again) and add some documentation. I'll try to look at testing FACT a bit as well.

[flibitijibibo]

Sounds good! For clarity's sake, we should probably start thinking about how the patchset should be organized. I'm thinking this, but if it could be broken down further that'd be fine too:

1. Fix submix stage sorting
2. DecodeCallback parameter changes
3. XAudio2: Add WMA support
4. XACT: Add WMA support

XACT in particular is a little bit fussy, so these can be sent out whenever they work, doesn't have to be all at once. For example, I'd be up for patches 1+2 right away, so we can test that with the existing decoders and trim the diff size down for 3 in case anyone that wants to look at WMA specifically, then I can look at 4 separately if needed since pretty much nobody except for me and Johan ever sees XACT WMA data these days...

[JohanSmet]

Yes, splitting these up is a good idea. I'll make PRs for the first two shortly.

[flibitijibibo]

This just got fixed by #46.