Audio

[MrBrixican]

Just wanted to know whether audio support is a planned addition to this library. Audio is arguably a necessary component of most games, and I think it's something that's generally expected of most game frameworks. If you are interested, I have a feature writeup:

---

When it comes to cross platform game audio libraries/wrappers, the big names are essentially OpenAL(Soft), SDL_Mixer, SoLoud, FMOD, Wwise, and BASS. The first two are pretty dated, and the latter three are commercially licensed, which I don't think fits the theme of this repository. SoLoud is great, but I had a few concerns (see note 1).

Thus, I suggest a relatively thin wrapper around cute_sound. I like it for the following reasons:

• hooks into sdl for non standard platforms
• single header (technically two with ogg vorbis), very lightweight
• has instanced audio (which allows control after an instance has been played)
• has the ability to get/set current sample position (very important for rhythm games, haha)
• simple music fade support

The following implementation details would be pretty important

• separate namespace (~Foster.Framework.Audio) and optionally enabled (via App.Run), to avoid conflicts if a user decides to use another audio library (see note 2)

I know you used cute_sound with blah, so your opinion would obviously mean more than these superficial observations.

If this is something that is interesting to you, I'd be more than happy to take an initial crack at it.

note 1: SoLoud is an extremely honorable mention. It's pretty weird to build, but I have managed it. It's also a very fully featured library with effects, buses, and 3d spatialization, which may or may not be overkill. However, the effect on binary size would probably be considerable (it's actually pretty light < 1MB, forgot I published more than just the binary as an artifact) and it would most definitely constitute a dependency.

note 2: A one size fits all approach will simply not meet the complex needs of all games. Similar to animation, sprite, and ECS implementations, the correct audio implementation largely depends on the use case. If we cover 90-99% of simple 2D use cases with a lightweight abstraction and don't get in a user's way if they decide to use something different, that would be good enough.

[MrBrixican]

miniaudio is more mature than I remember it being. It's high level (engine) api seems like it could be a good fit. Seems like the only component that's a bit iffy is having to init a sound per instance and then deiniting once it's no longer in use (what defines that would be up to us). Also not sure about ogg support.

[NoelFB]

Yeah I definitely want some form of Audio support & agree with everything you've written up here. The main reason I don't have it yet is I 1) know very little about how the internals work and 2) work with audio teams that usually want to use existing proprietary tools like FMOD.

But that said I do want something simple and open source that can be used for small games. I am definitely on the same page regarding note 2 - Trying to implement a large scale generic system is not something I want to try to maintain, especially when people will likely just not use it anyway if they need more advanced features.

So that said, I do feel like a wrapper around cute_sound or miniaudio makes sense, similar to how the platform API currently wraps stb_image/stb_truetype. I also agree it should be in its own namespace, but probably still exist in the Framework folder (so Foster.Framework.Audio, like you suggested).

If you're interesting in taking a swing at this I'd be more than happy to give feedback and merge that in once ready. I'm not super sure whether cute_sound or miniaudio makes more sense, although I do agree I like both of these over soloud just due to their single-file nature.

[NoelFB]

Oh also it might make sense to let App.Run take a flags parameter, for various initialization flags such as disabling audio. I'm not sure what else it would have yet, though.

[MrBrixican]

Sounds good, I'll work on this later this week. I'll probably do an initial implementation with cute_sound just to get something quickly working, but I have a feeling that miniaudio will be better featured if I can get instancing implemented correctly.

[amerkoleci]

Sounds good, I'll work on this later this week. I'll probably do an initial implementation with cute_sound just to get something quickly working, but I have a feeling that miniaudio will be better featured if I can get instancing implemented correctly.

I would go with miniaudio, as it offers more stuff than cute_audio, or FAudio as it integrates greatly with SDL

[MrBrixican]

I agree with you on miniaudio, it would probably be the ideal long term solution. However, FAudio retains a lot of bloat in order to maintain full XNA compatability (not saying that's bad but not the same requirements we have).

[NoelFB]

Yeah I am in favor of miniaudio I think. I also feel like FAudio is a great project but it falls into the "larger audio" systems for me, and I feel like I want to try to keep the core of this framework as light as possible. If someone wants to use that instead they can just disable our default audio implementation.

[amerkoleci]

Sounds good, at this point I would suggest to turn off unused subsystems from SDL (like audio, renderer and probably others) this way resulting library is smaller

[NoelFB]

That's a good idea! I can probably control those with CMake compile flags somehow.

[MrBrixican]

I have been messing around with miniaudio and must say, it's a fantastic library. It should be pretty trivial to create some sound management Platform methods that are very thin layers over what is already built in.

I do have a couple of thoughts I'd like to cover before I begin my implementation:

---

Sound Instancing

miniaudio has a very minimal implementation of sound instancing. For the most part, it is very similar to OpenAL in that you must manually create and manage sounds (equivalent to OpenAL sources). In order to play the same audio multiple times concurrently, you would need to instantiate multiple sounds. Fire and forget audio would obviously require a layer that sits above miniaudio. Additionally, by default the resource manager will deallocate resources based on reference counts, so if you uninit all sounds pointing to an audio file, it will unload the data and further dispatches of the same audio would incur a file load.

Manual sound management is an unfortunate burden to place on the user, who should probably not be required to have knowledge of these inner workings. Thus, I will create a set of classes/structs that will provide a further abstraction for audio, similar to how Batch abstracts rendering.

Mainly, it will allow for loading an audio file one time, and then being able to create sound instances that can optionally be manipulated. An instance will automatically be disposed of behind the scenes when it stops playing.

---

Scope

I will aim to support basic sound features:

• play
• pause
• stop
• volume
• pitch
• pan
• current position / play time
• length
• looping

I may add spatialization, fading, and scheduled stop/start seeing as they're already implemented in miniaudio.

I will not be adding:

• Effects
  • The node based effect system in miniaudio is pretty complex and this would add a significant amount of complexity to an already complex first pass
• Microphone support
  • Most games don't use mic input, so this is not a priority
• Directly writing pcm data to output device
  • Again, most games don't require this functionality

---

Misc

The docs and issues made note of potential issues with ogg vorbis files (specifically get length methods), however I didn't notice any issues at all (possible my test files are exceptions).

There are some intricacies around the config settings you can use for the built in resource manager and the effect that has on runtime memory/cpu usage. However, those are easily configured so I'll leave that for the final pass.

---

If you have any thoughts you'd like to share, please chime in.

[NoelFB]

Hey this all sounds super good to me, thanks for working on this! I'm on the same page regarding implementation and features.

Various thoughts:

• I was thinking it might be worth making it possible to exclude the audio stuff from the native lib under an optional flag in case you want to run this on a weird platform that miniaudio doesn't support, but it's probably not necessary. I assume miniaudio just won't do anything on platforms it can't handle.
• I am also definitely in favor of handling the memory/instancing internally. I think the use case of this API should definitely favor simplicity and easy of use. If people want more complicated audio they can use a big external library.
• One consideration, depending on how this is set up, is trying to avoid GC allocations when playing new sound instances. If possible I could see the API working more like you get a struct handle back when playing a new instance, instead of allocating a class object. I'm not sure how realistic this is, but it would be nice to avoid allocating C# objects every time the user plays a sound.

[MrBrixican]

Regarding unsupported platforms, miniaudio defaults to a null backend which simply does nothing, as you say. Optionally, if it's a really weird platform miniaudio can be configured to back onto sdl. That should really only be consoles though, since miniaudio supports emscripten, desktop, and mobile platforms out of the box.

I 100% agree on the GC allocations, struct handles were what I had in mind, as it was something that bothered me with the XNA implementation. There will be plenty of unavoidable c allocations, but those will not touch the .NET GC.

[NoelFB]

Yeah, that makes sense to me! Sounds great.

[Flip120]

Hi, I created a very basic audio c# wrapper for miniaudio to be included in my foster game, some things still to be implemented/improved like instance management and adding more features but I guess this is more or less the approach you were discussing about using handles isn't it? https://github.com/Flip120/miniaudio_csharp/tree/main

Here the platform part: https://github.com/Flip120/miniaudio_csharp/blob/main/platform/audio_api.h https://github.com/Flip120/miniaudio_csharp/blob/main/platform/audio_api.c

The c# wrapper: https://github.com/Flip120/miniaudio_csharp/blob/main/C_Lib_Test/AudioPlatform.cs

And a little test program: https://github.com/Flip120/miniaudio_csharp/blob/main/C_Lib_Test/Program.cs

[MrBrixican]

Apologies, I haven't updated this thread with the latest.

I made a miniaudio wrapper to be used with Foster with the following:

• Automated builds for Linux/Mac/Windows
• Supported features
  • Loading WAV, MP3, FLAC, QOA, OGG, and raw PCM data
  • Essential operations/settings: Play, Pause, Stop, Seek, Volume, Pitch, Pan, Looping, Spatialization
  • Sound groups to manage multiple sound instances (useful for sound category volume management)
  • Garbage free managed sound instances

There's still a bit more I want to do with effects and writing audio to file (likely breaking changes), but it's pretty well rounded for now. Also included is a small music visualizer example:

https://github.com/MrBrixican/Foster.Audio

[heavyrain266]

Hm, while I like the idea for built-in audio interface, mayby we should consider opaque interface of sorts? I agree with Noel on the fact that we should keep this framework lightweight, and audio is rather controversial topic with lots of different opinions.

In case of opaque audio, Foster could provide an Audio class for use in third party extensions to provide swappable playback engines to the end users.

[MrBrixican]

The problem is that even playback is opinionated. For example, FMOD uses the concept of events, which can actually be a collection of sounds and effects that can organized in a timeline based fashion. Whereas miniaudio and other low level audio libraries work more on the concept that a sound is just one specific sound being played at a specific time.

Due to how opinionated Audio can be, the most Foster can do to abstract Audio is provide life cycle hooks (Startup, Update, Shutdown) which it already does through Modules.

After talking with Noel privately, the intent is for my wrapper to be separate (not built in) from Foster Framework as an option to those who don't have more complex audio use cases.

[RandyGaul]

I wrote cute_sound.h and would be happy to prototype a lightweight wrapper implementation. It's recently got some updates for high-perf pitch modulation, sort of rounding out the last major feature I can think you'd want for 2D games that was missing. cute_sound isn't much code, is high perf, and provides a handle based API which is super easy to extend to other languages.

miniaudio while full-featured is really a lot of code and very complex. I'd say more appropriate for a large game with dedicated sound department or an audio studio with a lot of recording or niche needs. It's a little more difficult to design a lightweight API when wrapping a library like miniaudio, and for frameworks like the kind Noel creates lightweight and paired to essentials is the name of the game.

cute_sound would hook into SDL2 directly and implements it's own high perf SIMD mixer, and falls back to scalar float ops mixer for full cross-platform support (define CUTE_SOUND_SCALAR for e.g. emscripten or ARM builds).

It's not a lot of code and provides the essentials for sound FX and music in 2D games such as

• load wav/ogg files, query audio sample rate/sample count/channel count
• play/pause/loop/pan/pitch/offset sound FX
• global controls for fx/music separately, and overall global controls
• music track player API with fade in/out or crossfade, great for jukebox style or sequences of tracks
• modulatable sound instances, easy lifetime management (i.e. nearly none)
• easy to pitch modulate sound FX as they play, or slow down/rewind music dynamically
• id based API for instances

Here's a sketch of the API as I've wrapped it in my own C++/Lua framework, just to give you an idea of what the full picture could look like on the .h header end:

CF_INLINE Audio audio_load_ogg(const char* path) { return cf_audio_load_ogg(path); }
CF_INLINE Audio audio_load_wav(const char* path) { return cf_audio_load_wav(path); }
CF_INLINE Audio audio_load_ogg_from_memory(void* memory, int byte_count) { return cf_audio_load_ogg_from_memory(memory, byte_count); }
CF_INLINE Audio audio_load_wav_from_memory(void* memory, int byte_count) { return cf_audio_load_wav_from_memory(memory, byte_count); }
CF_INLINE void audio_destroy(Audio audio) { cf_audio_destroy(audio); }
CF_INLINE void audio_cull_duplicates(bool true_to_cull_duplicates = false) { cf_audio_cull_duplicates(true_to_cull_duplicates); }
CF_INLINE int audio_sample_rate(Audio audio) { return cf_audio_sample_rate(audio); }
CF_INLINE int audio_sample_count(Audio audio) { return cf_audio_sample_count(audio); }
CF_INLINE int audio_channel_count(Audio audio) { return cf_audio_channel_count(audio); }

// -------------------------------------------------------------------------------------------------

CF_INLINE void audio_set_pan(float pan) { cf_audio_set_pan(pan); }
CF_INLINE void audio_set_global_volume(float volume) { cf_audio_set_global_volume(volume); }
CF_INLINE void audio_set_sound_volume(float volume) { cf_audio_set_sound_volume(volume); }
CF_INLINE void audio_set_pause(bool true_for_paused) { cf_audio_set_pause(true_for_paused); }

// -------------------------------------------------------------------------------------------------

CF_INLINE void music_play(Audio audio_source, float fade_in_time = 0) { cf_music_play(audio_source, fade_in_time); }
CF_INLINE void music_stop(float fade_out_time = 0) { cf_music_stop(fade_out_time); }
CF_INLINE void play_music(Audio audio_source, float fade_in_time = 0) { cf_music_play(audio_source, fade_in_time); }
CF_INLINE void stop_music(float fade_out_time = 0) { cf_music_stop(fade_out_time); }
CF_INLINE void music_set_volume(float volume) { cf_music_set_volume(volume); }
CF_INLINE void music_set_loop(bool true_to_loop) { cf_music_set_loop(true_to_loop); }
CF_INLINE void music_set_pitch(float pitch = 1.0f) { cf_music_set_pitch(pitch); }
CF_INLINE void music_pause() { cf_music_pause(); }
CF_INLINE void music_resume() { cf_music_resume(); }
CF_INLINE void music_switch_to(Audio audio_source, float fade_out_time = 0, float fade_in_time = 0) { cf_music_switch_to(audio_source, fade_out_time, fade_in_time); }
CF_INLINE void music_crossfade(Audio audio_source, float cross_fade_time = 0) { cf_music_crossfade(audio_source, cross_fade_time); }
CF_INLINE void music_set_sample_index(int sample_index) { cf_music_set_sample_index(sample_index); }
CF_INLINE int music_get_sample_index() { return cf_music_get_sample_index(); }

// -------------------------------------------------------------------------------------------------

CF_INLINE Sound sound_play(Audio audio_source, SoundParams params = SoundParams()) { return cf_play_sound(audio_source, params); }
CF_INLINE Sound play_sound(Audio audio_source, SoundParams params = SoundParams()) { return cf_play_sound(audio_source, params); }

CF_INLINE bool sound_is_active(Sound sound) { return cf_sound_is_active(sound); }
CF_INLINE bool sound_get_is_paused(Sound sound) { return cf_sound_get_is_paused(sound); }
CF_INLINE bool sound_get_is_looped(Sound sound) { return cf_sound_get_is_looped(sound); }
CF_INLINE float sound_get_volume(Sound sound) { return cf_sound_get_volume(sound); }
CF_INLINE float sound_get_pitch(Sound sound) { return cf_sound_get_pitch(sound); }
CF_INLINE int sound_get_sample_index(Sound sound) { return cf_sound_get_sample_index(sound); }
CF_INLINE void sound_set_is_paused(Sound sound, bool true_for_paused) { cf_sound_set_is_paused(sound, true_for_paused); }
CF_INLINE void sound_set_is_looped(Sound sound, bool true_for_looped) { cf_sound_set_is_looped(sound, true_for_looped); }
CF_INLINE void sound_set_volume(Sound sound, float volume) { cf_sound_set_volume(sound, volume); }
CF_INLINE void sound_set_pitch(Sound sound, float pitch = 1.0f) { cf_sound_set_pitch(sound, pitch); }
CF_INLINE void sound_set_sample_index(Sound sound, int sample_index) { cf_sound_set_sample_index(sound, sample_index); }
CF_INLINE void sound_stop(Sound sound) { cf_sound_stop(sound); }

I'm totally open to adjusting the design to fit in with Foster-style aesthetics and could provide continued support as well whenever needed. I'm happy to put together a PR and get going if Noel feels like sharing his thoughts and likes the idea.

[MrBrixican]

I really enjoyed cute_sound! I'd be a fan even if it was standalone, haha. C# has many good libraries, but very few simple to use cross platform audio libraries. The more the merrier.

Regarding miniaudio, I think the main barrier of entry currently is that many of the more useful features are locked behind having to manage sound instance state (allocating, deallocating, filters, handles, etc.) yourself. Once you implement that, it is relatively simple to use. However, I won't argue it isn't heavier or more complex than cute_sound.

[RandyGaul]

Thanks for explaining the management situation. One thing I wanted to note from some experience maintaining these kinds of frameworks, having to include a system to integrate a third party library incurs a lot of risk. If I were considering a PR for some of my framework code that involved complex custom integration I'd have to probably pass on it. Incorporating code is a big risk and reducing maintenance cost in the long term is a priority.

[MrBrixican]

Completely agree. Less is more in many cases.