Implement own version of audio reverb effect

Frooxius commented 1 month ago

Is your feature request related to a problem? Please describe.

As part of move to a process based architecture (https://github.com/Yellow-Dog-Man/Resonite-Issues/issues/706) and eventual move away from Unity, we need to move majority of remaining functionality fully to FrooxEngine.

Part of this is a custom audio system with direct Steam Audio integration: https://github.com/Yellow-Dog-Man/Resonite-Issues/issues/2430

One of the components that this also needs are reverb zones.

Describe the solution you'd like

We'll need to implement/integrate the audio reverb effect directly into FrooxEngine, so we can compute this effect on our own, instead of relying on Unity.

Based on research, Unity is using FMod, which uses I3DL2 based reverb effect. An remake version of this effect exists here: https://github.com/vananasun/I3DL2Refurb

As part of this, we'll need to do following:

Investigate what other alternatives exist for this and what is their compatibility with the existing reverb zone (we'll need to preserve as much as possible)
Create a wrapper for the native library (if we find one)
Integrate with Resonite

Describe alternatives you've considered

We could license FMod and use it as the audio system, but this incurs additional costs on our end. Alternatively we could license WWise instead (which might be better in some ways), but this also has licensing costs in thousands of dollars, so might not be viable.
We could look into other free solutions as well (e.g. OpenAL), but I'm unsure if it provides compatible effects or if it would be a good fit
We could drop audio reverb zone for now, but this would break existing content
SteamAudio seems to offer some reverb effects on its own. We could investigate if those can be used to replicate the reverb zone and use it to emulate it

Additional Context

This is needed for this: https://github.com/Yellow-Dog-Man/Resonite-Issues/issues/2430

I'm creating this as a separate issue since it's a related, but distinct problem.

If you're aware of some good solutions in this space, suggestions are welcome!

Requesters

No response

Redd56 commented 1 month ago

a bit of a slightly out of the ordinary suggestion but i know airwindows has a reverb plugin. tho all of the plugins are made with being vst's in mind it might be neat to have a few (if not all) of the airwindows plugins as audio components. could lead to some very neat results

Redd56 commented 1 month ago

actualy theres quite a lot, chamber, creamcoat, crunchcoat, galactic, nonliniarspace, kcathedral3 and kplateA-D

JackTheFoxOtter commented 1 month ago

I'm most curious about how good the native Steam Audio reverb effect is compared to the current one in use by Resonite. Since the new audio system will have direct Steam Audio integration, it sounds like a potentially good choice to me, provided the quality of the effect is good enough.

Frooxius commented 1 month ago

It's less about how good it is and more about getting it sound similar to preserve old content.

Frooxius commented 1 month ago

Asked @BlueCyro to help with this once he's done with AVIF stuff.

There's few tasks that need to be done for this (in this order):

[ ] Do more research to see if there are any other solutions for this (e.g. checking if Steam Audio can replicate the effects of the reverb zone and seeing what other libraries are available)
[ ] Check if https://github.com/vananasun/I3DL2Refurb can replicate the reverb effect currently in Resonite
[ ] If above project can replicate those, separate the algorithm into a standalone DLL and create wrapper so we can run the processing

Frooxius commented 1 month ago

I've done a bit more research on this too. There's some more libraries:

Freeverb (https://github.com/sinshu/freeverb) - has only limited number of parameters, so very likely not suitable at all
Soundpipe (https://paulbatchelor.github.io/proj/soundpipe.html) - has several reverb modules - bigverb, jcrev and zitarev. -- zitarev might be suitable, it has a number of parameters: https://paulbatchelor.github.io/res/soundpipe/docs/zitarev.html
https://github.com/sukh-sekhon/Convolution-Reverb - didn't check what parameters it supports
https://github.com/HiFi-LoFi/KlangFalter - has a good number of parameters, but would need to test

BlueCyro commented 2 weeks ago

So I've been working on this for a while and I think I have a pretty good implementation, but I wanna gather some feedback on a few things.

The current candidate for replacement is soundpipe's Zita reverb, which has a good number of parameters that seem to be able to at least get within a ballpark of what our current reverb zone sounds like.

Here are some examples (headphones highly recommended):

Bathroom

Zita: https://github.com/user-attachments/assets/99045295-fb58-43ad-bf85-1137bcfb70dd

Unity: https://github.com/user-attachments/assets/8c67a4e6-3724-4417-9aaf-dd3f3baf7770

Hangar

Zita: https://github.com/user-attachments/assets/3e129e8e-fffc-4fd0-af44-e50cbf5fb878

Unity: https://github.com/user-attachments/assets/2d9a4d29-a8a0-4a41-94d3-438655eed09b

StoneCorridor

Zita: https://github.com/user-attachments/assets/64deac00-337b-434f-985d-5a5d6fb00def

Unity: https://github.com/user-attachments/assets/7026c7f5-ff01-4f43-8843-4522beb10672

Psychotic

Zita: https://github.com/user-attachments/assets/321d8679-e0fe-4e36-9698-e0ae916a0993

Unity: https://github.com/user-attachments/assets/e111baeb-3fc3-4d22-a56f-f5c423ade836

Some of them do in fact sound different (notably, they lack a sort of 'room' effect in some cases that I can't replicate), but it seems about acceptable to my ears.

Now, due to the stark difference in the amount of parameters and what they control, this presents a difficulty with upgrading the current reverb while retaining its sound profile. One solution is to use machine-learning (ML.NET is what was suggested), but after some research this doesn't appear as easy as the site makes it seem. Particularly, this is due to the fact that the models seem incapable(?) of multi-label classification (e.g. outputting multiple values instead of a single confidence score) which means I'd effectively have to train 12 models to match the number of presets that Zita's reverb has.

Another solution is simply doing a "best-guess" approach, wherein the upgrade process will decide "These values look most like this preset, so I'll just set it to this preset". E.x: If your custom reverb values are closest to "StoneCorridor", the upgrade process will just choose that preset, or if it's closest to "Psychotic", it will choose that one. This is obviously less accurate, but better than no conversion at all.

The machine-learning route appears to be a struggle-bus-and-a-half, and I want to gauge whether that's the only real solution, or if any better ideas are viable that aren't as complex.

So I want some feedback/suggestions from the community here: Would it be worth pursuing a machine-learning solution? How important is it to you that the reverb sound exactly, mostly, or even just kind of the same once it's upgraded? Is the reverb seldom-used enough that the best-guess solution would be viable? Please let me know what you think.

Suggestions on possible alternative solutions to the conversion are welcome.

Also, here's the preset mappings so you can see how each preset is mapped to Zita numerically: ZITA_PRESET_MAPPINGS.json

lxw404 commented 2 weeks ago

Personally, I don't think having it match exactly is important whatsoever, as long as the newer solution is actually good. It would probably be a huge time sink to even consider matching the parameters for ultimately not much fidelity difference, not to mention the very small percentage of content that actually makes use of reverb in the first place.

It will be tedious to do multivariate multi-output regression on this type of thing since you'll have to collect a ton of data for, imo, not much gain at all.

shiftyscales commented 2 weeks ago

If I could recommend Cyro- it'd probably be best to have a few standardized clips you test against- voice is a good starting point- but ultimately it'd be used for environments with music, ambient sound effects and other kinds of things too since reverb has different effects at different sound frequencies- having a standardized set of clips also helps more easily A/B test the two audio solutions. Even still- I can definitely hear some pretty stark differences in some of the presets you've set up above- specifically the "bathroom" and "psychotic" ones.

That also said- I am also of the opinion that as long as things are "close enough" it shouldn't ultimately matter a ton- as mentioned above- from my time on Resonite- I have seen a fairly minimal use of reverb zones- probably the most extensive use I've seen are on some of the VibeZ maps, and in Kulza's worlds.

Matching them so things are 'close enough' and A/B testing in some prominent Resonite worlds that actually feature reverb zones should be sufficient I think.

Similar to the upgrade to the linear colourspace- it should be expected there will be some minor differences- with this upgrade- as long as things are 'close enough' it shouldn't matter too much otherwise- especially if the reverb zones can be tweaked sufficiently by users after the fact.

BlueCyro commented 2 weeks ago

I'll see about running through some of the less accurate adaptations of these effects with a wider variety of sound sources. I don't actually have the reverb hooked up to the zones (right now it's just a mod that affects my audio stream), but I'll see if I can hook it up such that I can switch between them and A/B the few worlds that use them.

Yellow-Dog-Man / Resonite-Issues