Audio mixer discussion

[inactive123]

Original post by @leiradel:

"My problem with the mixer was that I couldn't decide how to store PCM data in memory. The problem is that the data comes in 16-bit signed integers, the resampler wants floats, the mixer wants 16-bit signed integers (I'm not sure about this one), and the audio sample rate can change at any time (as I was told at the dev channel.)

So if I keep audio data as int16_t, I'll need to convert to float and resample every time a sound is played. I can convert to float only once and resample every time, but then we'll need twice the memory for audio data. I wanted to get away with resampling every time, but then I'd need to keep the original data around so if the sample rate changes I could resample to the new rate."

[inactive123]

Also, here is a member on the libretro forums - it would be worthwhile to take his requests/needs onboard while we design this audio mixer -

https://forums.libretro.com/t/audio-engine-to-use-in-cores/6718/3

[inactive123]

@leiradel Alcaro's response so far is:

"Twice the memory for audio data doesn't matter for in-flight data like resampler output. It's small already, 2*16KB-or-whatever isn't worth worrying about"

[Alcaro]

I also said "95% of that issue is way beyond my qualifications".

If resampler is run continuously and immediately sent to mixer and speaker, then resampler output is small and insignificant. If resampler is run on large buffers at once, then resampler output could be big and worth caring about.

I don't know what the proposed usecase is (I don't even know if you want resampler before or after mixer, or both), so I can't say anything certain.

[inactive123]

The usecases for this audio mixer will be:

• Libretro cores
• RetroArch itself (we should be able to mix custom soundtrack music and the libretro core's music together, as well as menu sounds)

Also -

• We have moved DSP audio filters now to libretro-common, and it is the expectation that each 'voice/stream' will be able to have a DSP audio filter applied to it on a per-stream basis if so desired.
• The usability for libretro cores will be - if some game needs something like SDL Audio Mixer, we can instead use our Libretro-common audio mixer instead.

[leiradel]

Retroluxury has a sound data object, which is the original PCM data that is loaded from a WAV file, and a sound object, which is the sound data converted to int16_t stereo samples, and resampled to a target sample rate. It only mixes sound objects, not sound data, because the mixing buffer is in the target sample rate. (The mixing buffer has to be in a known rate, so it's better to make it the same rate as the output device and avoid another resample.)

Libretro cores RetroArch itself (we should be able to mix custom soundtrack music and the libretro core's music together, as well as menu sounds)

Libretro cores would probably want to convert upfront and discard the original data, since the output rate is known, but RetroArch allows the output rate to change, so maybe we should keep the unconverted data around to resample if needed.

Twice the memory for audio data doesn't matter for in-flight data like resampler output. It's small already, 2*16KB-or-whatever isn't worth worrying about

Maybe I'm worrying too much, but on the other hand I don't know what the impact on platforms like NDS and PSP would be. Maybe keeping the original sound data on those platforms would consume precious bytes that could be used elsewhere.

Anyway, to support both RetroArch and libretro cores, I think we should keep the original sound data around to support changes to the output rate (and on a second thought, the core could also change it's output sample rate with an env call.) It's really the memory consumption that worries me.

Also, here is a member on the libretro forums - it would be worthwhile to take his requests/needs onboard while we design this audio mixer -

Retroluxury's audio mixer already has the foundation for all that. It loads and resamples WAV files, and plays OGG files resampling on the fly, so all one has to do is to create other voices supporting other decoders.

[frranck]

Would be nice to have some .Mod or .Sid support.

[frranck]

Also seems v2m is the new cool format, it sounds pretty good.

[RobLoach]

Thanks to leiradel for introducing RetroArch's Audio Driver/Mixer recently. Has support for OGG and WAV files currently.

I'd love to see it...

• Used for background music/sounds in the menu
• Moved into libretro-common so that we could use parts of it in other cores like Lutro
• Have support for more formats, like @frranck suggested

[leiradel]

Would be nice to have some .Mod or .Sid support.

It shouldn't be hard to add support for other music formats, the problem are the dependencies that we must add to RetroArch to decode the files. See SDL_mixer for an example.

Moved into libretro-common so that we could use parts of it in other cores like Lutro

There's an audio mixer in libretro-common already, I think it's used in Mr. Boom core. I think this one could be extended to support other formats since it's outside the RetroArch codebase.

[ghost]

Some notes franck:

• V2M support will need support for the libv2m 1.0 synth as well as 1.5. That would be a mess to do. Plus the working code is written in NASM.
• SID support would need decent things like ReSID. And ReSID is in C++
• MOD support is easy, there is code out there for MOD/S3M/XM that is accurate to the original player's source code.

[frranck]

@mudlord MOD would be perfect for me.

[frranck]

Not sure about the feasibility, but another nice option would be the possibility to stream background music directly from youtube. i.e. https://www.youtube.com/watch?v=c17k4LfLkaE

[ghost]

Looked at MP3 and FLAC decoding. That should round out the mainstream formats, since AAC is a hornet's nest of fuckery when it comes to patents.

Not sure about what to do next.

[frranck]

@twinaphex What's the current state for .MOD playing ? I think there was a bounty resolved at some point.

[RobLoach]

I could be wrong, but I think what's in there is WAV, mp3, FLAC, ogg, and mod. Unsure if IBX is compiled for your platform, but it is worth a shot.

@mudlord Got the mixer compiled in a core, but now getting a segfault when loading .WAV files. I'll try to split the code into its own core to help debugging. Do you still have that sample core around?

[frranck]

@RobLoach Thanks, I tried to load .MOD and .XM in that branch https://github.com/Javanaise/mrboom-libretro/tree/adding_music using ibxm but I'm getting some "audio_mixer_play_mod cannot retrieve duration !" errors.

[frranck]

If I change https://github.com/Javanaise/mrboom-libretro/blob/adding_music/libretro-common/audio/audio_mixer.c#L599 to replay = new_replay( module, 48000, 1 ); // ( module, s_rate, 1); I don't get the error message but still don't get the module played...

[ghost]

@RobLoach http://rebote.net/random/audio_playback_wav.rar

[RobLoach]

Put up that playback tester over at: https://github.com/RobLoach/libretro-common-audio-test

I've figured out how to load different audio formats, but am unable to be able to switch the position of the playback. Anyone know how to do that? I tried playing with .position, but that didn't do anything.

[ghost]

I'd imagine you need a seeking api for that.

[Javanaise]

FYI there's an audio related bounty on https://github.com/Javanaise/mrboom-libretro/issues/36

[ghost]

Does Mr Boom use retropad? Or does it need keyboard input support?

[BenMcLean]

I am interested in trying to make a Rock Band Unplugged style music game as a libretro core. That'll need real good audio mixing for sure.