Querying the soundspaces 2.0 simulator

[sreeharshaparuchur1]

Hi,

I would like to know if it is possible to experience the difference in simulator quality between soundspaces 1.0 and soundspaces 2.0 without having to explicitly train a model using soundspaces 2.0 and performing a qualitative analysis.

Or, how would you suggest going about creating a script to enable manual navigation and spectrogram visualization in this environment?

Thanks

[ChanganVR]

@sreeharshaparuchur1 by experiencing the difference do you mean listening to the sound and hearing the difference? SoundSpaces 2.0 renders more accurate reverberation level so you should be hearing louder reverberation in general compared to SoundSpaces 1.0. There is not much difference between SS 1 and SS 2 in terms of rendering quality besides this point. The main advantage of SS 2 would be its configurability, continuity and generalizability.

It's very easy to render some examples without creating a navigation agent. You could check out this example for a quick tutorial for SS 2: https://github.com/facebookresearch/sound-spaces/blob/main/examples/soundspaces2_quick_tutorial.ipynb

[sreeharshaparuchur1]

Hey,

Apologies for not getting back sooner. Yes, what I mean by 'experiencing the difference' is being able to navigate through the environment while being able to listen to the sound (the script you've provided helps in this regard, so thank you for that).

As mentioned in the soundspaces 2.0 paper, has the diffraction limitation of soundspaces 1 been solved completely? If not, to what extent is diffraction modeled? I'd be grateful if you could provide a 'pressure map' of sound intensities in a scene similar to what you have done in soundspaces 1.0 with the following figure: Clearly, there is a limitation in how sound is perceived within the vicinity of occlusions such as walls (as visible in the region right outside the door where the sound source is located as well as how the pressure wave remains in more or less a straight line from the room in which the sound source is present.)

Lastly, despite following the instructions in the readme and this issue, I am still unable to run the notebook that you've pushed as I run into this error:

AttributeError                            Traceback (most recent call last)
/tmp/ipykernel_453290/3383711455.py in <module>
     37 audio_sensor_spec.uuid = "audio_sensor"
     38 audio_sensor_spec.enableMaterials = True
---> 39 audio_sensor_spec.channelLayout.type = habitat_sim.sensor.RLRAudioPropagationChannelLayoutType.Binaural
     40 audio_sensor_spec.channelLayout.channelCount = 1
     41 # audio sensor location set with respect to the agent

AttributeError: 'habitat_sim._ext.habitat_sim_bindings.RLRAudioPropagationChannelLayout' object has no attribute 'type'

I have looked at this issue, however, I do not understand how to 'update the binaries'. I'd appreciate your support on this.

[ChanganVR]

Hi @sreeharshaparuchur1, unfortuantely I don't have time these days to work on getting the pressure map for you due to my time constraint. But it should be fair doable as you could sample a grid of points and render for these pairs of points, and based on which you can compute the pressure map.

How we approach diffraction has been described in our papers with references to my collaborator Carl's papers. He would be the best person to ask about the details if you have further questions after reading this paper.

As for the error, looks like you still buildtthe package incorrectly. And you should use this commit for the build: https://github.com/facebookresearch/habitat-sim/commits/RLRAudioPropagationUpdate. If you were using main earlier, that was the wrong branch.

[sreeharshaparuchur1]

Hey, I understand the time constraints of work, no an issue. I just wanted to know if you had generated one during your testing phase for the simulator. I will use the approach that you've mentioned to create the map, thank you.

However, in order to proceed with generating said pressure map, I'd need the demo codes that you have pushed to run. Unfortunately, I do not have the below mentioned files

"data/scene_datasets/mp3d/mp3d.scene_dataset_config.json"
"data/scene_datasets/gibson/gibson_semantic.scene_dataset_config.json"

Kindly share them so that I and everyone else would be able to use the demo codes.

Lastly, the attribute error was down to using the 'main' branch of habitat-sim, I had to use this as a separate issue that I raised in the habitat-sim repo only pushed the updated changes to the main branch. However, as of today, the RLRAudioPropogationUpdate branch does have that change and I was able to build the simulator and run the above code (until the part where I need the scene config files) without any error.

[ChanganVR]

Hi @sreeharshaparuchur1 it's great to know it's working for you now! These configs files are provided here: https://github.com/facebookresearch/habitat-sim/blob/main/DATASETS.md#gibson-and-3dscenegraph-datasets (SceneDatasetConfig files).

[sreeharshaparuchur1]

Thanks for linking to that repo and telling me where to find the required json files @ChanganVR ! I should've asked about this in the previous comment itself but I forgot to :grimacing: - where can I find:

"data/mp3d_material_config.json"

as used in:

audio_sensor.setAudioMaterialsJSON("data/mp3d_material_config.json")

And would this file work even for the Gibson dataset or any dataset in general (as SS2.0 should be compatible universally)?

[ChanganVR]

@sreeharshaparuchur1 the material config file is here: https://github.com/facebookresearch/rlr-audio-propagation/blob/main/RLRAudioPropagationPkg/data/mp3d_material_config.json. I tested on MP3D/Gibson/HM3D and it's working pretty well. Although there are some differences in how objects are named across datasets, they are overall similar.

[sreeharshaparuchur1]

Hahaha, I found the file at that location by searching for it in the habitat-sim repo (counterintuitive, but one unrelated issue was able to load the file successfully as indicated by the error log which displayed the path to the material config file) just as you replied to my issue. The example notebook and script now work!!

Thanks for all your help @ChanganVR !

P.S. although, i'm still not sure where you pulled 'singing.wav' from since it isn't in the set of sounds that are downloaded via this command :stuck_out_tongue:

I will close the issue though XD

[ChanganVR]

@sreeharshaparuchur1 the singing sound is here: https://github.com/facebookresearch/sound-spaces/blob/main/res/singing.wav :)