FosterFramework / Foster

A small C# game framework
MIT License
450 stars 37 forks source link

Audio #19

Open MrBrixican opened 1 year ago

MrBrixican commented 1 year ago

Just wanted to know whether audio support is a planned addition to this library. Audio is arguably a necessary component of most games, and I think it's something that's generally expected of most game frameworks. If you are interested, I have a feature writeup:


When it comes to cross platform game audio libraries/wrappers, the big names are essentially OpenAL(Soft), SDL_Mixer, SoLoud, FMOD, Wwise, and BASS. The first two are pretty dated, and the latter three are commercially licensed, which I don't think fits the theme of this repository. SoLoud is great, but I had a few concerns (see note 1).

Thus, I suggest a relatively thin wrapper around cute_sound. I like it for the following reasons:

The following implementation details would be pretty important

I know you used cute_sound with blah, so your opinion would obviously mean more than these superficial observations.

If this is something that is interesting to you, I'd be more than happy to take an initial crack at it.

note 1: SoLoud is an extremely honorable mention. It's pretty weird to build, but I have managed it. It's also a very fully featured library with effects, buses, and 3d spatialization, which may or may not be overkill. However, the effect on binary size would probably be considerable (it's actually pretty light < 1MB, forgot I published more than just the binary as an artifact) and it would most definitely constitute a dependency.

note 2: A one size fits all approach will simply not meet the complex needs of all games. Similar to animation, sprite, and ECS implementations, the correct audio implementation largely depends on the use case. If we cover 90-99% of simple 2D use cases with a lightweight abstraction and don't get in a user's way if they decide to use something different, that would be good enough.

MrBrixican commented 1 year ago

miniaudio is more mature than I remember it being. It's high level (engine) api seems like it could be a good fit. Seems like the only component that's a bit iffy is having to init a sound per instance and then deiniting once it's no longer in use (what defines that would be up to us). Also not sure about ogg support.

NoelFB commented 1 year ago

Yeah I definitely want some form of Audio support & agree with everything you've written up here. The main reason I don't have it yet is I 1) know very little about how the internals work and 2) work with audio teams that usually want to use existing proprietary tools like FMOD.

But that said I do want something simple and open source that can be used for small games. I am definitely on the same page regarding note 2 - Trying to implement a large scale generic system is not something I want to try to maintain, especially when people will likely just not use it anyway if they need more advanced features.

So that said, I do feel like a wrapper around cute_sound or miniaudio makes sense, similar to how the platform API currently wraps stb_image/stb_truetype. I also agree it should be in its own namespace, but probably still exist in the Framework folder (so Foster.Framework.Audio, like you suggested).

If you're interesting in taking a swing at this I'd be more than happy to give feedback and merge that in once ready. I'm not super sure whether cute_sound or miniaudio makes more sense, although I do agree I like both of these over soloud just due to their single-file nature.

NoelFB commented 1 year ago

Oh also it might make sense to let App.Run take a flags parameter, for various initialization flags such as disabling audio. I'm not sure what else it would have yet, though.

MrBrixican commented 1 year ago

Sounds good, I'll work on this later this week. I'll probably do an initial implementation with cute_sound just to get something quickly working, but I have a feeling that miniaudio will be better featured if I can get instancing implemented correctly.

amerkoleci commented 1 year ago

Sounds good, I'll work on this later this week. I'll probably do an initial implementation with cute_sound just to get something quickly working, but I have a feeling that miniaudio will be better featured if I can get instancing implemented correctly.

I would go with miniaudio, as it offers more stuff than cute_audio, or FAudio as it integrates greatly with SDL

MrBrixican commented 1 year ago

I agree with you on miniaudio, it would probably be the ideal long term solution. However, FAudio retains a lot of bloat in order to maintain full XNA compatability (not saying that's bad but not the same requirements we have).

NoelFB commented 1 year ago

Yeah I am in favor of miniaudio I think. I also feel like FAudio is a great project but it falls into the "larger audio" systems for me, and I feel like I want to try to keep the core of this framework as light as possible. If someone wants to use that instead they can just disable our default audio implementation.

amerkoleci commented 1 year ago

Sounds good, at this point I would suggest to turn off unused subsystems from SDL (like audio, renderer and probably others) this way resulting library is smaller

NoelFB commented 1 year ago

That's a good idea! I can probably control those with CMake compile flags somehow.

MrBrixican commented 1 year ago

I have been messing around with miniaudio and must say, it's a fantastic library. It should be pretty trivial to create some sound management Platform methods that are very thin layers over what is already built in.

I do have a couple of thoughts I'd like to cover before I begin my implementation:


Sound Instancing

miniaudio has a very minimal implementation of sound instancing. For the most part, it is very similar to OpenAL in that you must manually create and manage sounds (equivalent to OpenAL sources). In order to play the same audio multiple times concurrently, you would need to instantiate multiple sounds. Fire and forget audio would obviously require a layer that sits above miniaudio. Additionally, by default the resource manager will deallocate resources based on reference counts, so if you uninit all sounds pointing to an audio file, it will unload the data and further dispatches of the same audio would incur a file load.

Manual sound management is an unfortunate burden to place on the user, who should probably not be required to have knowledge of these inner workings. Thus, I will create a set of classes/structs that will provide a further abstraction for audio, similar to how Batch abstracts rendering.

Mainly, it will allow for loading an audio file one time, and then being able to create sound instances that can optionally be manipulated. An instance will automatically be disposed of behind the scenes when it stops playing.


Scope

I will aim to support basic sound features:

I may add spatialization, fading, and scheduled stop/start seeing as they're already implemented in miniaudio.

I will not be adding:


Misc

The docs and issues made note of potential issues with ogg vorbis files (specifically get length methods), however I didn't notice any issues at all (possible my test files are exceptions).

There are some intricacies around the config settings you can use for the built in resource manager and the effect that has on runtime memory/cpu usage. However, those are easily configured so I'll leave that for the final pass.


If you have any thoughts you'd like to share, please chime in.

NoelFB commented 1 year ago

Hey this all sounds super good to me, thanks for working on this! I'm on the same page regarding implementation and features.

Various thoughts:

MrBrixican commented 1 year ago

Regarding unsupported platforms, miniaudio defaults to a null backend which simply does nothing, as you say. Optionally, if it's a really weird platform miniaudio can be configured to back onto sdl. That should really only be consoles though, since miniaudio supports emscripten, desktop, and mobile platforms out of the box.

I 100% agree on the GC allocations, struct handles were what I had in mind, as it was something that bothered me with the XNA implementation. There will be plenty of unavoidable c allocations, but those will not touch the .NET GC.

NoelFB commented 1 year ago

Yeah, that makes sense to me! Sounds great.

Flip120 commented 7 months ago

Hi, I created a very basic audio c# wrapper for miniaudio to be included in my foster game, some things still to be implemented/improved like instance management and adding more features but I guess this is more or less the approach you were discussing about using handles isn't it? https://github.com/Flip120/miniaudio_csharp/tree/main

Here the platform part: https://github.com/Flip120/miniaudio_csharp/blob/main/platform/audio_api.h https://github.com/Flip120/miniaudio_csharp/blob/main/platform/audio_api.c

The c# wrapper: https://github.com/Flip120/miniaudio_csharp/blob/main/C_Lib_Test/AudioPlatform.cs

And a little test program: https://github.com/Flip120/miniaudio_csharp/blob/main/C_Lib_Test/Program.cs

MrBrixican commented 7 months ago

Apologies, I haven't updated this thread with the latest.

I made a miniaudio wrapper to be used with Foster with the following:

There's still a bit more I want to do with effects and writing audio to file (likely breaking changes), but it's pretty well rounded for now. Also included is a small music visualizer example:

https://github.com/MrBrixican/Foster.Audio

heavyrain266 commented 7 months ago

Hm, while I like the idea for built-in audio interface, mayby we should consider opaque interface of sorts? I agree with Noel on the fact that we should keep this framework lightweight, and audio is rather controversial topic with lots of different opinions.

In case of opaque audio, Foster could provide an Audio class for use in third party extensions to provide swappable playback engines to the end users.

MrBrixican commented 7 months ago

The problem is that even playback is opinionated. For example, FMOD uses the concept of events, which can actually be a collection of sounds and effects that can organized in a timeline based fashion. Whereas miniaudio and other low level audio libraries work more on the concept that a sound is just one specific sound being played at a specific time.

Due to how opinionated Audio can be, the most Foster can do to abstract Audio is provide life cycle hooks (Startup, Update, Shutdown) which it already does through Modules.

After talking with Noel privately, the intent is for my wrapper to be separate (not built in) from Foster Framework as an option to those who don't have more complex audio use cases.

RandyGaul commented 2 months ago

I wrote cute_sound.h and would be happy to prototype a lightweight wrapper implementation. It's recently got some updates for high-perf pitch modulation, sort of rounding out the last major feature I can think you'd want for 2D games that was missing. cute_sound isn't much code, is high perf, and provides a handle based API which is super easy to extend to other languages.

miniaudio while full-featured is really a lot of code and very complex. I'd say more appropriate for a large game with dedicated sound department or an audio studio with a lot of recording or niche needs. It's a little more difficult to design a lightweight API when wrapping a library like miniaudio, and for frameworks like the kind Noel creates lightweight and paired to essentials is the name of the game.

cute_sound would hook into SDL2 directly and implements it's own high perf SIMD mixer, and falls back to scalar float ops mixer for full cross-platform support (define CUTE_SOUND_SCALAR for e.g. emscripten or ARM builds).

It's not a lot of code and provides the essentials for sound FX and music in 2D games such as

Here's a sketch of the API as I've wrapped it in my own C++/Lua framework, just to give you an idea of what the full picture could look like on the .h header end:

CF_INLINE Audio audio_load_ogg(const char* path) { return cf_audio_load_ogg(path); }
CF_INLINE Audio audio_load_wav(const char* path) { return cf_audio_load_wav(path); }
CF_INLINE Audio audio_load_ogg_from_memory(void* memory, int byte_count) { return cf_audio_load_ogg_from_memory(memory, byte_count); }
CF_INLINE Audio audio_load_wav_from_memory(void* memory, int byte_count) { return cf_audio_load_wav_from_memory(memory, byte_count); }
CF_INLINE void audio_destroy(Audio audio) { cf_audio_destroy(audio); }
CF_INLINE void audio_cull_duplicates(bool true_to_cull_duplicates = false) { cf_audio_cull_duplicates(true_to_cull_duplicates); }
CF_INLINE int audio_sample_rate(Audio audio) { return cf_audio_sample_rate(audio); }
CF_INLINE int audio_sample_count(Audio audio) { return cf_audio_sample_count(audio); }
CF_INLINE int audio_channel_count(Audio audio) { return cf_audio_channel_count(audio); }

// -------------------------------------------------------------------------------------------------

CF_INLINE void audio_set_pan(float pan) { cf_audio_set_pan(pan); }
CF_INLINE void audio_set_global_volume(float volume) { cf_audio_set_global_volume(volume); }
CF_INLINE void audio_set_sound_volume(float volume) { cf_audio_set_sound_volume(volume); }
CF_INLINE void audio_set_pause(bool true_for_paused) { cf_audio_set_pause(true_for_paused); }

// -------------------------------------------------------------------------------------------------

CF_INLINE void music_play(Audio audio_source, float fade_in_time = 0) { cf_music_play(audio_source, fade_in_time); }
CF_INLINE void music_stop(float fade_out_time = 0) { cf_music_stop(fade_out_time); }
CF_INLINE void play_music(Audio audio_source, float fade_in_time = 0) { cf_music_play(audio_source, fade_in_time); }
CF_INLINE void stop_music(float fade_out_time = 0) { cf_music_stop(fade_out_time); }
CF_INLINE void music_set_volume(float volume) { cf_music_set_volume(volume); }
CF_INLINE void music_set_loop(bool true_to_loop) { cf_music_set_loop(true_to_loop); }
CF_INLINE void music_set_pitch(float pitch = 1.0f) { cf_music_set_pitch(pitch); }
CF_INLINE void music_pause() { cf_music_pause(); }
CF_INLINE void music_resume() { cf_music_resume(); }
CF_INLINE void music_switch_to(Audio audio_source, float fade_out_time = 0, float fade_in_time = 0) { cf_music_switch_to(audio_source, fade_out_time, fade_in_time); }
CF_INLINE void music_crossfade(Audio audio_source, float cross_fade_time = 0) { cf_music_crossfade(audio_source, cross_fade_time); }
CF_INLINE void music_set_sample_index(int sample_index) { cf_music_set_sample_index(sample_index); }
CF_INLINE int music_get_sample_index() { return cf_music_get_sample_index(); }

// -------------------------------------------------------------------------------------------------

CF_INLINE Sound sound_play(Audio audio_source, SoundParams params = SoundParams()) { return cf_play_sound(audio_source, params); }
CF_INLINE Sound play_sound(Audio audio_source, SoundParams params = SoundParams()) { return cf_play_sound(audio_source, params); }

CF_INLINE bool sound_is_active(Sound sound) { return cf_sound_is_active(sound); }
CF_INLINE bool sound_get_is_paused(Sound sound) { return cf_sound_get_is_paused(sound); }
CF_INLINE bool sound_get_is_looped(Sound sound) { return cf_sound_get_is_looped(sound); }
CF_INLINE float sound_get_volume(Sound sound) { return cf_sound_get_volume(sound); }
CF_INLINE float sound_get_pitch(Sound sound) { return cf_sound_get_pitch(sound); }
CF_INLINE int sound_get_sample_index(Sound sound) { return cf_sound_get_sample_index(sound); }
CF_INLINE void sound_set_is_paused(Sound sound, bool true_for_paused) { cf_sound_set_is_paused(sound, true_for_paused); }
CF_INLINE void sound_set_is_looped(Sound sound, bool true_for_looped) { cf_sound_set_is_looped(sound, true_for_looped); }
CF_INLINE void sound_set_volume(Sound sound, float volume) { cf_sound_set_volume(sound, volume); }
CF_INLINE void sound_set_pitch(Sound sound, float pitch = 1.0f) { cf_sound_set_pitch(sound, pitch); }
CF_INLINE void sound_set_sample_index(Sound sound, int sample_index) { cf_sound_set_sample_index(sound, sample_index); }
CF_INLINE void sound_stop(Sound sound) { cf_sound_stop(sound); }

I'm totally open to adjusting the design to fit in with Foster-style aesthetics and could provide continued support as well whenever needed. I'm happy to put together a PR and get going if Noel feels like sharing his thoughts and likes the idea.

MrBrixican commented 2 months ago

I really enjoyed cute_sound! I'd be a fan even if it was standalone, haha. C# has many good libraries, but very few simple to use cross platform audio libraries. The more the merrier.

Regarding miniaudio, I think the main barrier of entry currently is that many of the more useful features are locked behind having to manage sound instance state (allocating, deallocating, filters, handles, etc.) yourself. Once you implement that, it is relatively simple to use. However, I won't argue it isn't heavier or more complex than cute_sound.

RandyGaul commented 2 months ago

Thanks for explaining the management situation. One thing I wanted to note from some experience maintaining these kinds of frameworks, having to include a system to integrate a third party library incurs a lot of risk. If I were considering a PR for some of my framework code that involved complex custom integration I'd have to probably pass on it. Incorporating code is a big risk and reducing maintenance cost in the long term is a priority.

MrBrixican commented 2 months ago

Completely agree. Less is more in many cases.