search

libretro / libretro-lutro

An experimental lua game framework for libretro following the LÖVE API
https://lutro.libretro.com
MIT License
149 stars 46 forks source link

Audio: Use Soloud? #187

Closed RobLoach closed 3 years ago

RobLoach commented 3 years ago

@jstine35 @kivutar I've seen some good momentum on the audio-front. I was playing around with using soloud a couple years ago. Not sure how far I got with demonstrating its use in a libretro core, but here you go: https://github.com/RobLoach/soloud/pull/1

May be worth investigating if maintaining the ogg/wav code is too much.

jstine35 commented 3 years ago

The main blocker is that (I believe) kivutar would prefer to avoid C++ compiler dependencies. @kivutar - please correct me if I'm wrong. I realize Soloud has a C API wrapper around a DLL, but that wouldn't solve the problem of porting Lutro to devices on which the Soloud DLL itself has not been ported.

As for evaluating current audio state and whether we want to try to go 3rdparty in general. Here's the list of things missing from Lutro currently (excluding purely API things like the Decoders):

We can do pitch control in-house. I've written a half dozen such mixers over the years. It's a day's job, including the usual things like interpolation and LPF that are usually needed to ensure good quality at the higher and lower spectrum of pitches.

Doppler is just a mathematical pitch adjustment, based on relative velocity of listener to producer. This could be implemented in lua, so long as pitch is implemented. Update rate wouldn't be as smooth as if it's baked into a mixer, but I doubt it would matter to the listener -- a vast majority of game console and AAA games have implemented such systems on framerate-ticked updates, eg 60hz.

Attenuation would benefit from proper filters to remove attenuated frequencies. I'm not sure if Love2D actually implements such things, or just uses lazy volume control to simulate it (seems more likely).

3D Positional Audio

If we are interested in 3D Positional Audio support, then absolutely it _is a must that we use a 3rdparty lib. If you don't do 3D positional audio right, the results are no better -- actually much worse -- than just doing some cheap clever stereo panning tricks. And doing it right is the very definition of non-trivial. For that reason I wouldn't blindly trust a 3rdparty lib to have done it correctly. The math is tricky and non-obvious because neither human hearing nor the amplitude of sound coming from speakers are linear. And there are issues around trying to pan sound that comes from directly in front, behind, to the sides of the listeners, or especially anywhere above or below. Speaker arrangements don't place speakers there, and there's only one central subwoofer, and the center front speaker is attenuated for speech and produces unexpected frequencies on other music and sndfx. It's a hot mess.

You can't really test or verify it's working right unless you have a process that uses waveform analysis on outputted audio. The majority of humans have very poor hearing and even with really good hearing it's hugely subjective as to whether something is positioning correctly. And then your own brain will get in your way after a while, "auto-correcting" for you thing that you hear if you work on it for long periods of time, to help it match what you're seeing on a screen. Finally, 3D positional audio usually also depends on environment: calculating impact of obstructions and the echo of surfaces. These things actually make the game feel more "immersive" than just spinning some sound around the player's 7 speakers.

These are just a few of the reason why entities like Wwise and FMOD have dedicated a significant amount of their developer time over the last decade to implementing quality 3D positional audio.

Furthermore, headphones are the predominant way in which people today listen to computer audio. So there goes the baby and the bath water too, unless you want to subscribe to the idea of micro-managing sub-wavelet timing that somehow rattles the bones below the skull before/after the bones of the ear and thus "simulates" the "sensation" of sound coming in from alt. angles.

3D positional audio summary: I have no interest in personally in positional audio. I'm not even sure Love2D's API makes a lot of sense about it, and who knows what positional audio impl it uses. And if whatever 3D positioning lib we use doesn't match Love2D's then we could end up with a surprisingly different result anyway.

RobLoach commented 3 years ago

Ah, didn't know soloud was C++. You're right to avoid that to keep the dependency chain as minimal as possible. The other two libraries I've seen thrown around a bit are miniaudio and cute_sound.

We're in a great state now! Thanks so much for all the help on pushing ogg forwards :wink: .... Positional audio would be cool, but you're right that it's not a high priority.