flibitijibibo
commented
1 year ago Those end up being pretty heavy-duty in terms of dependencies - FAudio is in the same boat as SDL where we really need everything to be either self-contained or something that's zlib-licensed (or stb-style public domain), since many targets can't bring something like ffmpeg into the mix. But, being in the same boat, I've kept a close eye on how SDL has dealt with resampling in particular, and the result is that we have 2 options:
- Add a path for libsamplerate, can be dynamically loaded: https://github.com/libsdl-org/SDL/blob/main/src/audio/SDL_audiocvt.c#L497
- Make resampling an FAudioplatform* function, and add pitch shift support to SDL_AudioStream: https://github.com/libsdl-org/SDL/blob/main/include/SDL3/SDL_audio.h#L685
Way back in the day I attempted to use option 2 and the result was pretty limited; we have to be able to adjust the sample rate pretty much all the time, so most resampling libraries don't cut it in terms of all the fussy things XAudio needs. I did my best to take notes from Chris Robinson and OpenAL Soft (without actually using the code, for licensing reasons), but yeah, it's definitely got some sharp edges.
Mixing is a bit tougher because of the way XAudio2 processes channels; mixing isn't hard but writing handwritten SIMD can be sometimes... unlike resampling I actually don't think there's anything that does exactly what XAudio does, but at the same time it's much easier to get correct than resampling.
zfigura
While trying to fix an issue related to unaligned buffers, I ran into some complications related to the use of EXTRA_DECODE_PADDING. While I believe it should be possible to modify our resampler to avoid the need for that, it's not exactly trivial, and it seems like less effort to simply outsource that effort by making use of existing code, leading me to wonder if this is something that has been considered before.
Obviously any new external library is a burden for distribution. On the other hand—not to put too fine a point on it—the current code seems complex and brittle, and this has deteriorated over time (a7703491, a756af4, eb94a810). The EXTRA_DECODE_PADDING also seems broken in a couple of ways, mostly by virtue of not handling end-of-buffer conditions.
Besides that, an external library would potentially offer better-quality resampling options (i.e. anything more sophisticated than the current simple linear interpolation), as well as SIMD support for the mixer, and more updated SIMD support for the resampler (e.g. GStreamer has a path for SSE 4.1). And of course there are the usual benefits of relying on an actively developed library—increased support for new features, more attention to bug fixes.
In summary, it seems well worth it to replace our resampler and mixer with an external library.
As for which external library to use—options that occur to me immediately are GStreamer and ffmpeg (aka libav). I haven't researched others.
GStreamer has a very simple synchronous API for resampling (libgstaudio) and a less simple callback-based API for mixing (i.e. it'd essentially require building a pipeline, though this isn't too hard, and I believe it can actually be done without callbacks if we care).
ffmpeg seems to be in about the same boat, although I'm not very familiar with its API. One downside is that it does not seem to commit to a stable or backwards-compatible API.
Is this something that would be acceptable? If so I'll start trying to implement this. If not I'd appreciate a link to discussion on the topic, if any.