Add Steam Audio integration

[sammonius]

Describe the project you are working on

A first-person shooter game where players need to be able to hear things like footsteps in 3D.

Describe the problem or limitation you are having in your project

The built-in 3D audio in Godot has no culling, echo, or any other environmental effects.

Describe the feature / enhancement and how it helps to overcome the problem or limitation

Steam open-sourced their audio library recently (early 2024) as apache 2. It could be integrated into the engine to replace the current 3D audio system. It supports a lot of environmental effects on audio, including echo, culling, and pass-through filters (like how wood normally doesn't let high-frequency sounds through as much as low-frequency ones). The library is ~4mb large according to the internet, though I can't verify this [EDIT: internet lied, it's 20mb].

Describe how your proposal will work, with code, pseudo-code, mock-ups, and/or diagrams

The AudioStreamPlayer3D node would use Steam Audio as the backend, and collision shapes could be used to cull the audio. Physics bodies would have an option to enable/disable audio culling, as well as an audio material (similar to physics materials) to control how audio is absorbed/reflected by it.

If this enhancement will not be used often, can it be worked around with a few lines of script?

nope, it would require a ton of GDExtension code.

Is there a reason why this should be core and not an add-on in the asset library?

It probably wouldn't be possible as an addon since it needs C++, and it improves a system that's already part of the engine.

[RedMser]

It could and likely should be a GDExtension to integrate Steam Audio and not part of Godot core. Although I don't think the audio system is currently extensible enough to make this a very seamless integration.

[AThousandShips]

I think this is best handled as a module or an extension, also please link to the library so it's easier to investigate

The built-in 3D audio in Godot has no culling, echo, or any other environmental effects.

Do you mean specifically spatial environmental effects? Because we do have echo and other environmental effects, and they can be controlled with Area3D, so please be a bit more specific in what this brings to the engine

The library is ~4mb large according to the internet, though I can't verify this.

This is pretty large if we're talking binary size, that adds about 6% to the binary size give or take (for Windows at least, with an optimized release binary of 65 mb at present)

[octanejohn]

related https://github.com/stechyo/godot-steam-audio/issues/18

[AThousandShips]

So this should be the Steam Audio library:

• https://github.com/ValveSoftware/steam-audio

But given their releases it looks like the library binaries are some 20 mb, unclear how much of this would be included if we included this library ourselves, linking it statically by embedding it, but even if it's just half of that it's enormous by comparison to the export projects

[octanejohn]

as gdextension is easier but it need this pr https://github.com/godotengine/godot/pull/86539

[AThousandShips]

How does it need that? What is missing? YOu can already do mixing?

[octanejohn]

@stechyo knows better its their pr also the steam-audio gdextension version

[stechyo]

The reason for that PR to exist is that, as far as I'm aware, there is no way to mix audio through Godot, then get the raw audio frames for subsequent processing, and make Godot play the processed audio. That PR is a proposal to do that, but ideally there would be a less hacky method of custom audio processing (if you want to see what kind of issues this PR creates, look no further than https://github.com/stechyo/godot-steam-audio/issues/39).

[AThousandShips]

You can mix audio through GDExtension, it's only not possible in GDScript, I'm not sure how the ability to get mixed audio from the stream has to do with this proposal?

[stechyo]

I was replying to the comment where I was tagged really, the PR is one version of getting SteamAudio to work with godot (expose that and do a gdextension, which I've done), but for there to be a more official integration I'd prefer something that was less prone to weird bugs.

[sammonius]

Sorry, when I made this proposal I thought GDExtension worked the same way GDNative did, and the engine needed to recompiled to use it. Since that's not true, this probably would be better as an addon.

(Should I close the thread or leave it open?)

[AThousandShips]

GDNative doesn't require recompiling either, only modules do, but this can be kept open if you still desire this feature in core, but some further details on what it'd actually bring and what kind of overhead or cost it'd bring would be important, if the 20 mb that seems to be the size of this library is the case for example, or even the low estimate of 4 mb

[sammonius]

It's probably best to close it since the dll is 20mb (the 4mb estimate was from a random google search and I assumed it was true for some reason). It would probably be best if this was written from scratch anyways so it could integrate with the godot physics and audio, instead of having its own systems and mixing between the two.

[AThousandShips]

It's possible that integrating it, and removing unnecessary parts of it (there might be backend code and drivers etc. included that we don't need), could bring the size down significantly, but it's a third party dependency which can take a lot of work to integrate, I haven't studied the code but it's possible a lot of work would have to be done to adjust it to our requirements

But a great first step would be to implement it as an extension, and work from there, or as a custom module, and if it works well it can be integrated either as an official add-on or into core

But at a glance the library has a lot of its own scene management stuff and seems very self contained in that way, I haven't dug deeper but I'm unsure how easy it would be to separate that and make it "just work" with the specific scene systems we use, it could prove a significant performance hurdle if we have to maintain a whole separate scene graph just inside the audio system to make it work

[fire]

I have experience doing this for resonance audio but the main thing to do is to reduce the engine patches needed.