search

Yellow-Dog-Man / Resonite-Issues

Issue repository for Resonite.
https://resonite.com
139 stars 2 forks source link

Integrate Steam Audio directly with FrooxEngine & with a custom audio system #2430

Open Frooxius opened 4 months ago

Frooxius commented 4 months ago

Is your feature request related to a problem? Please describe.

Currently the audio system used with FrooxEngine is a hybrid - the computation of all sound buffers for individual sound buffers happens within FrooxEngine, but those individual audio sources/outputs are then mixed & spatialized by Unity.

This makes us dependent on some Unity's audio system behaviors (e.g. only having a single listener) and also poses a challenge for multi-process architecture: https://github.com/Yellow-Dog-Man/Resonite-Issues/issues/706

Describe the solution you'd like

Implement custom audio mixing system directly in FrooxEngine & integrate Steam Audio directly for computing spatialization. This will allow FrooxEngine to compute the audio fully on its own into a single buffer, which is much easier to pass over an IPC or perhaps output directly.

This will also give us control to implement additional features, such as rendering multiple independent audio listeners, which would enable features like these:

Describe alternatives you've considered

We could make system that manages & proxies the audio buffer data over IPC, but this increases amount of data and complexity of implementation.

Additional Context

No response

Requesters

No response

charlie-sans commented 1 month ago

with the custom audio system, would we be able to get components or protoflux nodes to do things like TTS from a string? or components that allow us to provide filtering to ones audio source for ingame effects or radio chatter aswell?

this is more of a though at the moment based on what i know from the issues listed from #706

ProbablePrime commented 1 month ago

do things like TTS from a string

This should be a separate request.

filtering to ones audio source for ingame effects or radio chatter aswell

This should be a separate request. Although see #567 and #2708 , and in particular: https://discord.com/channels/1040316820650991766/1154514014563483759/1281532372583583789

charlie-sans commented 1 month ago

makes sense, ill go check those out now!