General questions about project acoustics

[AvivSham]

Hi, Thank you for this fascinating project, I have just watched the demonstration from 2019 and it looks amazing! Few questions I have in mind:

1. Do you plan to create a plugin for Blender as well?
2. I must use Azure services in order to use this package? Let's say I have a supercomputer can't I run the package there? i.e. the algorithm is not open sourced?
3. I consider using the package and I want to estimate the cost of it. What would be the cost of running a simulation of a single room with 5-10 sound sources and 1 receiver? The environment is not complex (presence of few items). The receiver is static and the sound sources are dynamic (moving in the environment)
4. continue of question (3) - If I have 5-10 objects in one room environment and I want to generate different scenes. Each scene will involve different movement patterns for each object. For each scene, I would need to run the acoustic simulation once again? (because the sound sources are moving around differently than before?)

Thank you in advance!

[kegodin]

Hi @AvivSham ! Thanks for your interest in Project Acoustics. To answer your questions: 1-2) We don't currently have a plan to create a Blender plugin, and the public package must be used with Azure, or, on Unity, can be used on your local PC. In some cases we are able to make source available through a source licensing agreement, then you could make the necessary changes. Please let us know if that's of interest.

3-4) Rather than specifying a single listener location, before the bake you must specify the regions where listeners can be placed using a navigation mesh. After the bake the listener can move anywhere in this region and you can make this region arbitrarily large. Larger areas result in more 'probes', and simulation cost scales linearly with the number of probes. After baking, you can specify any source location in the scene and sources can move at runtime without a re-bake.

[MikeChemi]

To add to what @kegodin said, if you have a super computer, I'd highly recommend checking out our documentation on running bakes on your local PC from Unity. https://docs.microsoft.com/en-us/gaming/acoustics/unity-baking#Local-bake

[AvivSham]

Let me check if I understood you correctly. (1) - If I use this plugin as part of Unity I can use it on my local PC / Server and if I'm using Unreal I must use Azure's services. (3-4) - changing the motion pattern of the objects won't result in a re-bake since the navigation mesh takes these probabilities into account.

@MikeChemi shared documentation for baking on local PC, according to the doc your package supports Win/OS, do you support Linux as well?

[kegodin]

Yep, your understanding is correct.

Unfortunately no, we don't provide Linux support for local bakes.

[AvivSham]

Last question please, is it possible to control all bake's parameters using code? w/o GUI (I assume that is possible as rendering scene using code is possible)

[kegodin]

It's not a scenario we've looked into, but I think it's possible. Both the Unreal and Unity plugins come with source code for the plugin UI components. It should be possible be able to modify the plugins to control the bake binaries from code instead of driving them from the UI.

[AvivSham]

@kegodin @MikeChemi Thank you for your quick response and your patience in answering all my questions!

[kegodin]

Glad to help! Thanks again for your interest in Project Acoustics. Let us know how your project goes - and we're here if you have any other questions.

[nikunjragh]

@AvivSham About your question 3: If you have a static listener that only rotates but doesn't move, you only really need 1 probe. The nav mesh is meant solely for specifying where the listener can move, and these listener position possibilities are sampled with probes. For each probe, an entire volume is simulated within which the sound sources can fly anywhere.

So, you could make the nav mesh very small so you get one probe at or near the known listener location. Sources/objects could still move however they like. You could easily do this local on a single PC in 20-30 mins for the default simulation volume which is about a city block in size.

[AvivSham]

Cool! that is a pretty good estimation. when I define the nav mesh I can control the prob sampling step can't I? for instance if I have a room which is 1cm cubic sized but I want high resolution nav mesh I can sample a prob each 1mm? (just a use case I won't do that) Another question please, can I place multiple receivers in different locations when some of them are static and some dynamic? (I'm trying to understand what are the control options of this package)

[kegodin]

All receiver locations are essentially dynamic, since the possible receiver locations at runtime are only constrained by the placement of the nav mesh and the subsequent probe locations.

If we're talking about use cases with this level of fine detail, this is a good time for me to mention that while the underlying wave solver is highly accurate, the acoustic data quantization and probe placement algorithms are designed for video games (i.e.: low cost and very approximate). We've been able to verify accuracy of simulated impulse responses vs. measured (e.g. timing and strength of early reflections, RT60) in a couple scenes, but the public package doesn't provide access to the impulse responses. If you need access to this level of accuracy, source licensing of the wave solver is something we could discuss.

[AvivSham]

So there is no problem with designing multi receivers scene (let's say a person with two ears). As for ACC, I'm not planning to develop a saving life application so I don't think such ACC level is required.

[kegodin]

Yep! You can implement multiple receivers. For that use case I'd recommend getting the Unreal plugin. The Unity plugin that makes calls into the runtime DLL is closed source. The Unreal plugin that makes 'QueryAcoustics()' calls to get the acoustics results is distributed in source form.

[AvivSham]

We are interested in simulating the audio signals as received by multiple microphones. Specifically, the microphone can be mounted on a person's head (hearing devices ot hearables) and we need to take into account the scattering and diffraction of the head and torso of the hearing aid wearer, namely the head-related transfer function (HRTF), preferably personalized to a specific person. We have access to databases of HRTFs (e.g. CIPIC) and we wish to incorporate the HRTF into the acoustic propagation in the environment. We will soon have a new Robot with a microphone array and it will be very nice to include the Robot-related transfer function into the simulator. Moreover, we would like to verify that if the signals are captured by multiple microphones, the signals undergo a realistic acoustic propagation model, such that we can process the signals with our spatially-aware algorithms (beamformer, and similar). In simple words, we would like to verify that all the reflections' attenuation and phase information are correctly produced, in a way that can allow coherent processing.

[kegodin]

Thanks for the additional detail. Unfortunately you won't be able to get the detail you need to complete this kind of project by using the public packages. The public packages only return parameters such as RT60, occlusion, and reverberation level.

[AvivSham]

Another question please, when using multiple receivers will the output be with multiple channels (as the number of listeners)? or it will be coherent (embedded in the video, single channel)? Is there a possibility to output both options? (we need to post-process the output audio signals)

[kegodin]

The plugins don't currently support multiple simultaneous receivers. You'd have to get our plugin source code and modify it to output the format you need. Currently a single receiver (the game listener) outputs a stereo HRTF signal.

[AvivSham]

Is it possible to get access to the source code? I want to mention, this is for research/educational purposes only.

[kegodin]

We generally only license to game studios, under the Xbox Game Development Kit (GDK) licensing process. If you're interested in academic licensing you can contact Nikunj Raghuvanshi via email for more information. His address is shown at the top of this page: http://www.nikunjr.com/.

[AvivSham]

Of course, the best option would be to get the source code but if not can you make us a custom patch that supports multiple simultaneous receivers?

[kegodin]

We're not building custom features at this time. We're happy to hear any suggestions or requests, and when themes appear, we add features that can benefit many users.

[AvivSham]

@kegodin Another question, in one of your messages you said: "Currently a single receiver (the game listener) outputs a stereo HRTF signal." Since it is stereo single i.e. two microphones can I output 2 sounds waves? one for each microphone? (two channels)

[AvivSham]

@kegodin can you please answer my previous question? and which operating systems supported by project acoustics in Unity?

[kegodin]

There is only one microphone in the current implementation. The acoustics system and the HRTF renderer are decoupled. The acoustics system computes acoustic parameters for that single microphone location, and the HRTF renderer render those parameters simulating a human listener standing in that location.

Windows and MacOS are supported for development. Windows, MacOS, Android, and Xbox are supported for deployment.

[AvivSham]

So it is not possible to post process the output of the HRTF renderer as two separate channels as human ears?

[kegodin]

Not sure I follow. Yes, the two separate channels are two separate ear signals.

[AvivSham]

I'm trying to understand if I will be able to post process each channel individually let's say I want to increase the phase difference between the channels or something like that.

[kegodin]

The Project Acoustics Unity plugin sends audio to the Unity audio engine, which renders it out to the OS audio system. Within Project Acoustics you don't have access to the audio to process it before rendering.

[AvivSham]

I understand that the question is if it is possible to get access to the two channels post rendering.

[kegodin]

Possibly. I can think of two possible methods: record from the OS audio system loopback, or find a way to get Unity's audio engine to record from itself. You'd have to try these to verify one of them works.

[MikeChemi]

@AvivSham two more options for you: You can write your own Unity audio plugin and insert it into the mixer after the Project Acoustics mixer. This will let you do whatever you want with the audio. Alternatively, you may try building and modifying the Project Acoustics Unity plugin yourself. The reference source is available here: https://github.com/microsoft/spatialaudio-unity/tree/develop/Source/Spatializer/ProjectAcoustics Note that this method still requires the DLL dependencies that are only available via the Unity package on download center.

[joy-void-joy]

In some cases we are able to make source available through a source licensing agreement, then you could make the necessary changes. Please let us know if that's of interest.

Hi! I am very interested in making a port of Project Acoustic to Blender. What would be the proper way to apply for it?

[NoelCross]

@joy-void-joy it is great to hear that you are interested in Project Acoustics in the context of Blender. I'm not sure what all is involved with providing Blender plugins, but you should be able to review the Unreal editor plugin and spatializer plugin code as an example of how to use the native APIs for Project Acoustics. Another option would be to export models from Blender and import them into either Unity3d or Unreal Engine where the support for Project Acoustics already exists.

It would be great to learn more about how you plan on using Project Acoustics in the context of Blender. If you are interested, we can have a call to discuss your plans in more detail to help put you on a path for success.

[nikunjragh]

Hi @joy-void-joy - I'll add that for PC, our existing binaries shipped with the Unreal plugin (plus public docker images for the bake tools) are probably sufficient to do all the steps, such as baking the ACE file and loading it at runtime. Maybe start with coarse bakes so you don't have to think of cloud dependency. I guess for Blender, "render time" equates to "runtime" for games. One big advantage is for Blender you'd know the camera trajectory, so you could create a "nav mesh" that is just that trajectory which will probably yield far fewer probes (and sim time) than typical for gaming apps.

At render time, you'd do acoustic queries for the camera trajectory for each rendered frame, and then, to begin, you could just modulate the loudness of each sound based on the query to drive an occlusion value per emitter at each frame. This would already render fairly rich variations in the audio mix as you enter or leave buildings etc. Reverb could also be setup up later as mixing various reverbs in proportions driven by the queries.

I'll echo Noel that this is a pretty interesting application area, and if you make any headway let us know and we could provide tips.