Calculate reverb in real-time based on the environment of the player

[Hiradur]

OpenAL Soft already has EFX which is a powerful way to create immersive environments by adding reverb to ingame sounds. However, the reverb generated by EFX is based on a preset and not calculated in real-time based on the environment of the player. This has two disadvantages. First, the realism of the reverb effect entirely depends on the chosen preset. It's possible to make a small room sound like a huge cathedral so its degree of realism depends on the preset chosen by the developer. EFX effects are also position independent, i.e. if a player has a wall next to one ear he won't be able to hear the wall.

A solution to both problems is to calculate reverb for each sound source in real-time based on the geometry of the environment. It would not only increase realism, it would also e.g. allow blind and skilled people to navigate virtual environments simply through the use of echolocation (https://en.wikipedia.org/wiki/Human_echolocation ).

Examples: A few audio engines already implement geometry-based reverb but haven't seen much following so far (with the exception of A3D 2.0 which was quite widespread in the 90's). Aureal A3D 2.0: https://www.youtube.com/watch?v=-oSlbyLAksM GSOUND: https://www.youtube.com/watch?v=buU8gPG2cHI unnamed GPU based engine: https://www.youtube.com/watch?v=TXUTgEmnD6U RAYA: https://www.youtube.com/watch?v=05EL5SumE_E Nvidia VRWorks Audio: https://www.youtube.com/watch?v=A3Y2BSHxKHU

Is reverb calculated in real-time based on the environment of a player something OpenAL Soft should be able to do in the future? There is no doubt this would be a long-term goal.

Challenges:

• Performance: Calculating reverb based on geometry requires expensive algorithms similar to raytracing and spatial filtering on dozens of sound sources.
• Representation of the environment: All games need to pass the surrounding geometry of the player to OpenAL Soft in some form. This representation should be scalable and be able to handle rough abstractions of the environment and highly detailed representations to be future proof. Ideally there also is a way to pass on data about what materials objects are made of since this has an influence on how these objects reflect sound.

[leilei-]

However, the reverb generated by EFX is based on a preset and not calculated in real-time based on the environment of the player.

You aren't stuck to specific presets. You could allocate one effect and keep changing its properties. I recently tried to do something like this in this engine by tracing from the rendered position several times . Haven't committed it though since I was disappointed by rhe outcome because it would have inconsistent results between standing/crouching/jumping off the floor in the same room, and it had limited accounting of materials it traced (metal and liquid are the only consistent ones)

To make it faster would need a prebaked 3d grid of presets, or quantized 3d blocks by their differences, and you're practically reimplementing EAGLE at that point.

[kcat]

EFX effects are also position independent, i.e. if a player has a wall next to one ear he won't be able to hear the wall.

EFX's EAXReverb effect has a panning property for the early reflections, so if you want to pan them to be stronger in a specific direction (in 3D), you can set that. The late reverb can be panned too, so the two panning properties can let you have multiple reverb areas that are panned in specific directions around the listener (which can be kept up to date as the listener moves and turns).

You aren't stuck to specific presets. You could allocate one effect and keep changing its properties.

Right. In theory, you could continually test the environment's geometry and materials around the listener, and generate reverb and occlusion/obstruction properties in real-time based on the detected environment. While it won't be as good as modeling individual reflections, it would be more appropriate to a dynamic scene compared to simply selecting a preset.

That said, I have talked with someone before regarding geometric reverb. The general consensus seems to be that you'll only want to do the initial/early reflections with it and let a more standard reverb method handle the late reverb/tail, because the behavior of sound waves and the number of wave fronts in a diffuse or reverberant room is extremely demanding to realistically model.

In the case of OpenAL Soft, something I may end up doing is allow the app to specify a set of points for the listener, their global positions and material/filter properties (the app would be responsible for figuring out the points and their properties, since it would be better designed to actually handle its own geometry+materials and any ray tests for it). It could then set a mask on each source, specifying how much it contributes to each point. So essentially you'd find the reflection points that would have the most noticeable response to the listener, then measure how much each source can "see" each point, thus modeling the first set of bounces of each sound to the listener.

OpenAL Soft would then work out the necessary delays and filters to mix the source with to each point. The output of those points are then mixed in 3D and fed into an EFX-like late reverb processing stage.

In theory, you could also handle second- or third-level bounces this way, by supplying the direction of the last bounce, but giving it a distance and material/filter property that combines the two or three bounces, and masking the source based on the position of the first bounce.

[Hiradur]

You aren't stuck to specific presets. You could allocate one effect and keep changing its properties.

Clever idea. I'll take a look.

The general consensus seems to be that you'll only want to do the initial/early reflections with it and let a more standard reverb method handle the late reverb/tail, because the behavior of sound waves and the number of wave fronts in a diffuse or reverberant room is extremely demanding to realistically model.

Sounds like the right way to go. As processing power increases one can calculate higher orders of bounces and delay the EFX further for a more realistic model.

[Leadwerks]

You know, the same technique used for voxel cone tracing might work for sound. Voxel data can be downsampled easily, so it gives an approximation of coverage that we use for soft/blurry ray trace reflection, something that would take many individual raycasts to do otherwise.

So if you want to get a mostly-accurate indication of how solid the area around the listener is, that is pretty fast to do, and the whole thing is near real-time and can be processed on another thread and updated intermittently.

[Hiradur]

Project Acoustics is an interesting approach to real-time environmental audio. If I understand it correctly, audio maps are baked from geometry that provide DSP parameters for sample points that accurately describe the acoustics of their location. It's been used in Gears of War 4 and 5 as well as Sea of Thieves.

I think OpenAL Soft already provides the DSP functionality required for such an approach. The baking of audio maps would probably better be left to the game engine or some kind of middleware.