Implement audio bus overhaul in support of improved audio spatialization

[ellenhp]

Describe the project you are working on

Godot engine audio.

Describe the problem or limitation you are having in your project

See #4377

Describe the feature / enhancement and how it helps to overcome the problem or limitation

This is a "how" whereas #4377 is more of a "what".

This proposal would add a new category of audio bus called a Spatial bus. This bus type would be different in that it would apply all effects prior to mixing the audio together. It would also be allowed to contain Audio Effects that change depending on spatial characteristics of the sound being played.

Currently, the DSP graph for a simple game with a Distortion bus might look like this.

This has a number of problems. The most obvious is that some audio effects tend to destroy spatial cues from panning. I picked a distortion effect because it's especially bad about that when the drive is turned up very high. What makes more sense is this:

This way, the distortion effect can mangle the sound, then the mangled sound can be attenuated and panned appropriately, rather than the mangling being applied only to the panned final mix.

Although each instance of the Distortion bus is separate on the DSP graph, there would only be one distortion bus in the bus layout.

All the instancing is done by the engine. See: godot#60480

The HRTF audio effect and resource type stuff proposed in #4377 can live in a spatial bus. Since each spatial effect chain is tied to a specific playback, the effect can monitor the spatial location of the player and listener and adjust the HRTF impulse responses it's using as the angles change. Distance-aware filter effects could also be added to the engine to let people attenuate sound the way they prefer. Some users have reported wanting more flexibility with attenuation https://github.com/godotengine/godot/issues/23485

Describe how your proposal will work, with code, pseudo-code, mock-ups, and/or diagrams

Implementation is mostly done in https://github.com/godotengine/godot/pull/60480

If this enhancement will not be used often, can it be worked around with a few lines of script?

No

Is there a reason why this should be core and not an add-on in the asset library?

See #4377

[reduz]

My suggestion here is that, given this is not such a common case scenario, its probably better to not redo the bus system just for it.

Instead, I think it should be simpler to have an AudioStreamWithEffects resource that inherits AudioStream, and allows setting a stream and a number of effects. This then wraps the playback of the stream, adding extra custom effects to it. It solves the problem and allows to keep the current simplicity of the audio bus system.

[ellenhp]

The downside of that is that games that use HRTF spatializers won't be able to use the bus system at all, because applying e.g. a distortion effect after a HRTF will completely destroy all of the spatial cues. The nice thing about the bus overhaul I proposed is that users can put their current effects before the spatializer instead of after, keeping the spatial cues. That's not possible if the spatializer is tied to the AudioStream unfortunately, unless each individual AudioStream is configured with effects individually which is very tedious.

I like the idea of adding a small amount of complexity to the bus system to make sure it works for everyone, but it's a tradeoff obviously since it definitely is added complexity.

[Firepal]

Maybe it could be togglable per-project to preserve some sort of backwards-compatibility, e.g. "Process Effects Before Positionalization" checkbox

[sebastienwood]

I'm sorry if my question is not appropriate, I'm not sure I fully understood the extent of the proposed audio spatialization. Would it grant more capacity to the engine in order to implement a "wavetracing" algorithm for audio such as the one in Microsoft Project Acoustics ? (for reference: https://www.youtube.com/watch?v=MAMz9dSHU04)

[ellenhp]

This overhaul would support that kind of use case, but without buy-in from the rest of the audio team it isn't going to happen. I really want to come up with something that's low-impact enough that it's agreeable for everyone, but I'm just about out of ideas. If I remember right, Juan wants to keep the panning calculations in the audio nodes (at least in comparison to any proposal I've made so far). Keeping spatialization in the nodes blocks implementation of any other kind of spatialization, unless you add new nodes in a GDExtension. This isn't niche, so I want to come up with something that we can agree on. Lots of people use e.g. FMOD and WWise integrations, and without an engine fork you can't reuse the existing audio nodes for those. That's a big quality of life problem IMO because it locks users into a specific integration, and makes porting into or out of that integration very difficult.

I lurk in the audio channel on the Godot discord and people there constantly try to extend the audio system in ways that are currently impossible. It sucks that the answer I have to give them is "you need to fork the engine and rewrite most of the audio system to do that".