Using SAVi with SS2.0's continuous simulator ?

Hello again @ChanganVR

Just wanted to confirm whether or not it is possible to use the code in SAVi with the continuous version of the simulator ? I have been playing with the instructions in the SAVi sub-directory's README, but the scripts seem to require the presence of the binaural RIRs dataset, which is quite heavy.

Since I would prefer to use the continuous version of the sound space 2.0 environment, is there a way to do so while avoiding download those hefty datasets ?

Thanks a lot for your time.

By default SAVi uses the precomputed RIRs files but nothing stops it from working on SoundSpaces 2.0.

There are some adjustments you'll need to make though. The first one is that you can't use the pretrained location predictor. You can either retrain it on the SS 2.0 data or train it on the fly with the policy. Secondly, you might want to think of a way to improve the training efficiency as SS 2 runs slower than SS 1 and SAVi requires a significant amount of episodes for training, which is definitley an interesting topic for research improvement.

Also note that, if you train a policy on SS 2, you can't compare it to the benchmark numbers in SAVi, you'll need to train and compare with your own baseline.

Despite these hassle, I think there is a lot of space to explored for semantic audio-visual navigation in continuous space. The first one potential improvement comes from the dataset, the object-sound association in SAVi was very much limited by the discrete nature of the grid points. The continuous rendering will definitely open up space for the object-sound association and the scale of the dataset. And secondly, as I mentioned above, there is also a lot of room for improvement for the sample efficiency of the algo.

Let me know if you have more questions.

Hi again @ChanganVR Thank you very much for your sustained supported, and sorry for the late answer.

Duly noted. I also think using the continuous simulator with different objects category producing the corresponding sounds instead of the dataset is a much more natural and closer to real world scenario. I will look a little bit more into the SAVi code to see how I could potentially be adapted to the SS2.0 then.

I will close the issue for now, but I hope I can still reach out to you here in case I have additional questions, instead of opening yet another issue.

Again, thanks a lot for your time.

Sorry for troubling you again, @ChanganVR

I was trying to get the SAVi baselines scripts work mostly using the pre-trained weights provided in the repository. When running the python ss_baselines/savi/run.py --exp-config ss_baselines/savi/config/semantic_audionav/savi_pretraining.yaml --model-dir data/models/savi or the python ss_baselines/savi/run.py --exp-config ss_baselines/savi/config/semantic_audionav/savi.yaml --model-dir data/models/savi which I believe spin up instances of the SemanticAudioNav task I get the following error:

Traceback (most recent call last):
  File "/home/rousslan/random/rl/ss-hab-headless-py39/sound-spaces/ss_baselines/savi/run.py", line 144, in <module>
    main()
  File "/home/rousslan/random/rl/ss-hab-headless-py39/sound-spaces/ss_baselines/savi/run.py", line 107, in main
    trainer.train()
  File "/home/rousslan/random/rl/ss-hab-headless-py39/sound-spaces/ss_baselines/savi/ddppo/algo/ddppo_trainer.py", line 293, in train
    rollouts.observations[sensor][0].copy_(batch[sensor])
RuntimeError: The size of tensor a (8) must match the size of tensor b (128) at non-singleton dimension 1
Exception ignored in: <function VectorEnv.__del__ at 0x7fd07316bd30>
Traceback (most recent call last):
  File "/home/rousslan/random/rl/ss-hab-headless-py39/habitat-lab/habitat/core/vector_env.py", line 592, in __del__
    self.close()
  File "/home/rousslan/random/rl/ss-hab-headless-py39/habitat-lab/habitat/core/vector_env.py", line 463, in close
    write_fn((CLOSE_COMMAND, None))
  File "/home/rousslan/random/rl/ss-hab-headless-py39/habitat-lab/habitat/core/vector_env.py", line 118, in __call__
    self.write_fn(data)
  File "/home/rousslan/random/rl/ss-hab-headless-py39/habitat-lab/habitat/utils/pickle5_multiprocessing.py", line 62, in send
    self.send_bytes(buf.getvalue())
  File "/home/rousslan/anaconda3/envs/ss-hab-headless-py39/lib/python3.9/multiprocessing/connection.py", line 205, in send_bytes
    self._send_bytes(m[offset:offset + size])
  File "/home/rousslan/anaconda3/envs/ss-hab-headless-py39/lib/python3.9/multiprocessing/connection.py", line 416, in _send_bytes
    self._send(header + buf)
  File "/home/rousslan/anaconda3/envs/ss-hab-headless-py39/lib/python3.9/multiprocessing/connection.py", line 373, in _send
    n = write(self._handle, buf)
BrokenPipeError: [Errno 32] Broken pipe

To look more into I have independently instantiated the environments based on the savi/configs/semantic_audionav.yaml configuration, and it appears that the depth sensor data is generated with the shape [NUM_PROCESSES, 128, 128, 1,1], instead of the expected [NUM_PROCESSES, 128, 128, 1], which makes the script break.

Basically, I created the environments, obtained the first observations by calling envs.reset(), and those are the fields in the observation dict:

for obs_sensor, obs_value in observations[0].items():
    print(f"{obs_sensor} -> {obs_value.shape}")

depth -> (128, 128, 1, 1)
rgb -> (128, 128, 3)
audiogoal -> (2, 16000)
spectrogram -> (65, 26, 2)
category -> (21,)
pointgoal_with_gps_compass -> (2,)
pose -> (4,)
location_belief -> (2,)
category_belief -> (21,)

I have tried to find where the depth sensor data is generated before being returned by the environment, but could not find where exactly it is.

Do you happen to have an idea on how to fix the environment code to make it return data that matches the observation_space["depth"] shape of (128,128,1) ?

Also, I noticed that there is category and category_belief fields in the observation. I guess the category is the type of the target object that generates the sound the agent is supposed to follow, as in the SAVi paper, but what would the category_belief be ?

Thanks for your time.

Hello again.

Setting training of RL agents aside, I tried to get the environments running based on the savi.yaml config while specifying CONTINUOUS: True in said .yaml file. However, it seems to use the binaural_rirs dataset instead, as I cannot see any environment simulation process on the GPU, unlike with the AudioNav tasks

Do the SAVi environment support continuous simulation already ? Or does it require some additional work on the sourd-spaces / habitat-sim / habitat-lab ?

Best regards.

@dosssman sorry about the delay. Regarding your question about the depth image dimension, I'm not sure where this extra dimension comes from because you probably modified the script? But it should be an easy fix and you could simply remove the extra dimension in the simulator.py file.

category is the ground truth semantic class of the sound and category_belief is the model's prediction (belief) of the sound, which is a placeholder variable for storing the model prediction output. It's not essential to the task. It's just some hack to make SAVi work.

Setting the simulation environment to be continuous for SAVi probably does not work directy as it was coded for discrete environment. You will need to modify the configurations a bit. If you are clueless about what the modifications are, I could look into it in the coming weekend. Would this work for you? In the meantime, you could try to do it yourself and it will definitely help you get a deeper understanding of the code.

@ChanganVR Thanks for the answer. For now I have been using a workaround that removes the extra dimension as suggested. I will try to look a bit deeper into the simulator.py file where the depth observatiosn is generated.

Regarding making the simulator for SAVi continuous, I have attempted to match the savi task default config script here with that of the av_nav tasks here to incorporate support for continuous simulator. Not sure if those are the relevant line, but the adding the lines highlighted in red on the default.py from Av_nav config to the default.py of SAVi on the right seemed to be responsible for the use of continuous simulator:

I also mirrored the corresponding changes from the av_nave base config and task config YAML file to those of the SAVi task.

When running a training script with such modification, the verbosity suggest that the SAVi experiment script is attempting to use the continuous simulator. However, back when I tried to get some experiment going a few days ago, it threw up some ambiguous error.

Ironically, when I queued up the experiment with continuous set to True for the SAVi environment just now to get the error traceback again, it seems to be working so far. I will post an update if it happens again.

In the mean time, I would appreciate if you could look up a bit into that on your side, or if you have any additional insights about the changes required to use SAVi continuous simulator.

Thanks for your time.

Well, it seems that the problem was due to not having enough GPU memory when I tried to run SAVi with the continuous simulator task before. SAVi with 10 parallel envs is staking around 18G of GPU memory. When I tried last week there was around only 15G free on the GPU. But some experiments finished and there was around 20G left when I started the experiment again as mentioned in the comment above, and so far the experiment has not thrown the error I got last time despite running for longer.

Will do a final update in around 24 hours to make sure it still works after some time has passed.

Hello again.

After leaving the script running for around 8 hours, it ended up crashing with a segmentation fault, so I am not sure what to think about that:

lCoherence:1, meshSimplification:0, dumpWaveFiles:0, enableMaterials:1, writeIrToFile:0
[01:19:39:583015]:[Sensor] AudioSensor.cpp(50)::AudioSensor : [Audio]  Channel Layout :  channelType:3, channelCount:2
[01:19:39:583023]:[Sensor] AudioSensor.cpp(55)::AudioSensor : [Audio]  output directory not provided, will use /home/AudioSimulation
[01:19:39:583033]:[Sensor] AudioSensor.cpp(60)::AudioSensor : [Audio]  OutputDirectory :  /home/AudioSimulation
[01:19:39:583093]:[Sensor] AudioSensor.cpp(169)::setAudioMaterialsJSON : Set audio materials database to json file :  data/mp3d_material_config.json
[01:19:39:725056]:[Sensor] AudioSensor.cpp(81)::setAudioSourceTransform : [Audio]  Setting the audio source position :  Vector(50.0362, 1.60707, 18.1695) ]
[01:19:39:974825]:[Sensor] AudioSensor.cpp(90)::setAudioListenerTransform : [Audio]  Setting the agent transform : position [ Vector(16.6656, 1.5802, 23.0604) ], rotQuat[ Vector(1, 0, 2.45858e-05, 0) ]
[01:19:39:974889]:[Sensor] AudioSensor.cpp(278)::createAudioSimulator : [Audio]  Create audio simulator iteration:  0
[01:19:39:975326]:[Sensor] AudioSensor.cpp(114)::runSimulation : [Audio]  Running the audio simulator
[01:19:39:975344]:[Sensor] AudioSensor.cpp(119)::runSimulation : [Audio]  New initialization, will upload geometry and add the source at position :  Vector(50.0362, 1.60707, 18.1695)
[01:19:39:975359]:[Sensor] AudioSensor.cpp(126)::runSimulation : [Audio]  Loading semantic scene
[01:19:39:975368]:[Sensor] AudioSensor.cpp(293)::loadSemanticMesh : [Audio]  Loading semantic mesh
[01:19:46:497743]:[Sensor] AudioSensor.cpp(398)::loadSemanticMesh : [Audio]  Vertex count :  14069477 , Index count :  18120 , Material :  blinds , LastUpdate :  false
[01:19:46:498108]:[Sensor] AudioSensor.cpp(398)::loadSemanticMesh : [Audio]  Vertex count :  14069477 , Index count :  38226 , Material :  beam , LastUpdate :  false
[01:19:46:498494]:[Sensor] AudioSensor.cpp(398)::loadSemanticMesh : [Audio]  Vertex count :  14069477 , Index count :  232560 , Material :  tv_monitor , LastUpdate :  false
[01:19:46:499919]:[Sensor] AudioSensor.cpp(398)::loadSemanticMesh : [Audio]  Vertex count :  14069477 , Index count :  7590 , Material :  clothes , LastUpdate :  false
[01:19:46:501590]:[Sensor] AudioSensor.cpp(398)::loadSemanticMesh : [Audio]  Vertex count :  14069477 , Index count :  2014416 , Material :  lighting , LastUpdate :  false
[01:19:46:512326]:[Sensor] AudioSensor.cpp(398)::loadSemanticMesh : [Audio]  Vertex count :  14069477 , Index count :  2701404 , Material :  door , LastUpdate :  false
[01:19:46:556191]:[Sensor] AudioSensor.cpp(398)::loadSemanticMesh : [Audio]  Vertex count :  14069477 , Index count :  16538367 , Material :  misc , LastUpdate :  false
[01:19:46:718061]:[Sensor] AudioSensor.cpp(398)::loadSemanticMesh : [Audio]  Vertex count :  14069477 , Index count :  968469 , Material :  objects , LastUpdate :  false
[01:19:46:744479]:[Sensor] AudioSensor.cpp(398)::loadSemanticMesh : [Audio]  Vertex count :  14069477 , Index count :  11403030 , Material :  window , LastUpdate :  false
[01:19:46:866354]:[Sensor] AudioSensor.cpp(398)::loadSemanticMesh : [Audio]  Vertex count :  14069477 , Index count :  235524 , Material :  cabinet , LastUpdate :  false
[01:19:46:867226]:[Sensor] AudioSensor.cpp(398)::loadSemanticMesh : [Audio]  Vertex count :  14069477 , Index count :  1644 , Material :  sink , LastUpdate :  false
[01:19:46:867648]:[Sensor] AudioSensor.cpp(398)::loadSemanticMesh : [Audio]  Vertex count :  14069477 , Index count :  344244 , Material :  curtain , LastUpdate :  false
[01:19:46:870479]:[Sensor] AudioSensor.cpp(398)::loadSemanticMesh : [Audio]  Vertex count :  14069477 , Index count :  2240178 , Material :  chair , LastUpdate :  false
[01:19:46:876377]:[Sensor] AudioSensor.cpp(398)::loadSemanticMesh : [Audio]  Vertex count :  14069477 , Index count :  20472 , Material :  appliances , LastUpdate :  false
[01:19:46:878136]:[Sensor] AudioSensor.cpp(398)::loadSemanticMesh : [Audio]  Vertex count :  14069477 , Index count :  1925673 , Material :  table , LastUpdate :  false
[01:19:46:883357]:[Sensor] AudioSensor.cpp(398)::loadSemanticMesh : [Audio]  Vertex count :  14069477 , Index count :  43386 , Material :  furniture , LastUpdate :  false
[01:19:46:901332]:[Sensor] AudioSensor.cpp(398)::loadSemanticMesh : [Audio]  Vertex count :  14069477 , Index count :  7892847 , Material :  floor , LastUpdate :  false
[01:19:46:921961]:[Sensor] AudioSensor.cpp(398)::loadSemanticMesh : [Audio]  Vertex count :  14069477 , Index count :  107307 , Material :   , LastUpdate :  false
[01:19:46:945824]:[Sensor] AudioSensor.cpp(398)::loadSemanticMesh : [Audio]  Vertex count :  14069477 , Index count :  14172090 , Material :  ceiling , LastUpdate :  false
[01:19:46:983071]:[Sensor] AudioSensor.cpp(398)::loadSemanticMesh : [Audio]  Vertex count :  14069477 , Index count :  3783 , Material :  board_panel , LastUpdate :  false
[01:19:46:983246]:[Sensor] AudioSensor.cpp(398)::loadSemanticMesh : [Audio]  Vertex count :  14069477 , Index count :  96495 , Material :  counter , LastUpdate :  false
[01:19:46:984028]:[Sensor] AudioSensor.cpp(398)::loadSemanticMesh : [Audio]  Vertex count :  14069477 , Index count :  409017 , Material :  seating , LastUpdate :  false
[01:19:46:985439]:[Sensor] AudioSensor.cpp(398)::loadSemanticMesh : [Audio]  Vertex count :  14069477 , Index count :  100233 , Material :  void , LastUpdate :  false
[01:19:46:985807]:[Sensor] AudioSensor.cpp(398)::loadSemanticMesh : [Audio]  Vertex count :  14069477 , Index count :  43341 , Material :  sofa , LastUpdate :  false
[01:19:46:988849]:[Sensor] AudioSensor.cpp(398)::loadSemanticMesh : [Audio]  Vertex count :  14069477 , Index count :  1171878 , Material :  column , LastUpdate :  false
[01:19:46:992219]:[Sensor] AudioSensor.cpp(398)::loadSemanticMesh : [Audio]  Vertex count :  14069477 , Index count :  23268 , Material :  mirror , LastUpdate :  false
[01:19:46:992454]:[Sensor] AudioSensor.cpp(398)::loadSemanticMesh : [Audio]  Vertex count :  14069477 , Index count :  128607 , Material :  plant , LastUpdate :  false
[01:19:47:026144]:[Sensor] AudioSensor.cpp(398)::loadSemanticMesh : [Audio]  Vertex count :  14069477 , Index count :  14763747 , Material :  wall , LastUpdate :  false
[01:19:47:321226]:[Sensor] AudioSensor.cpp(398)::loadSemanticMesh : [Audio]  Vertex count :  14069477 , Index count :  65307 , Material :  picture , LastUpdate :  false
[01:19:47:321820]:[Sensor] AudioSensor.cpp(398)::loadSemanticMesh : [Audio]  Vertex count :  14069477 , Index count :  150723 , Material :  shelving , LastUpdate :  false
[01:19:47:322693]:[Sensor] AudioSensor.cpp(398)::loadSemanticMesh : [Audio]  Vertex count :  14069477 , Index count :  273552 , Material :  chest_of_drawers , LastUpdate :  false
[01:19:47:323733]:[Sensor] AudioSensor.cpp(398)::loadSemanticMesh : [Audio]  Vertex count :  14069477 , Index count :  46209 , Material :  stool , LastUpdate :  false
[01:19:47:323926]:[Sensor] AudioSensor.cpp(398)::loadSemanticMesh : [Audio]  Vertex count :  14069477 , Index count :  714 , Material :  towel , LastUpdate :  true
[01:20:22:700295]:[Sensor] AudioSensor.cpp(143)::runSimulation : [Audio]  Adding source at position :  Vector(50.0362, 1.60707, 18.1695)
[01:20:22:702291]:[Sensor] AudioSensor.cpp(151)::runSimulation : Running simulation, folder :  /home/AudioSimulation0
[01:20:22:714484]:[Sensor] AudioSensor.cpp(259)::getObservationSpace : [Audio]  getObservationSpace -> [ChannelCount] :  2 , [SampleCount] :  4740
[01:20:23:137666]:[Sensor] AudioSensor.cpp(90)::setAudioListenerTransform : [Audio]  Setting the agent transform : position [ Vector(68.0362, 1.48715, 20.1695) ], rotQuat[ Vector(0.707107, 0, 0.707107, 0) ]
[01:20:23:137734]:[Sensor] AudioSensor.cpp(278)::createAudioSimulator : [Audio]  Create audio simulator iteration:  1
[01:20:23:137753]:[Sensor] AudioSensor.cpp(114)::runSimulation : [Audio]  Running the audio simulator
[01:20:23:137764]:[Sensor] AudioSensor.cpp(151)::runSimulation : Running simulation, folder :  /home/AudioSimulation1
[01:20:23:140617]:[Sensor] AudioSensor.cpp(259)::getObservationSpace : [Audio]  getObservationSpace -> [ChannelCount] :  2 , [SampleCount] :  9035
./runs.sh: line 62: 297162 Segmentation fault      (core dumped) python ppo_savi.py --exp-name "ppo_savi_continuous" --total-steps 10000000 --wandb --wandb-project ss-hab --wandb-entity dosssman --logdir-prefix $LOGDIR_PREFIX --seed $seed

From the output above, I guess the continuous simulator is indeed working, but not stable enough.

@dosssman That's weird. I've encoutered segmentation fault in the past but it always happens at the beginning due to loading incorrect material files etc.

The only thing I could possibly think of is the simulator was loading one bad environment and for Matterport3D, I had to exlcude this environment: https://github.com/facebookresearch/sound-spaces/blob/a1f882f248da13bd9ef51c42cb0fe40a44f4e1d0/ss_baselines/common/env_utils.py#L46-L48

Could you check the longer log and see which environment was loaded?

@ChanganVR Thanks for the answer.

Scrolled up the log and the latest scene that is loaded seems to be the following one:

[01:18:40:072069]:[Metadata] MetadataMediator.cpp(66)::setSimulatorConfiguration : Set new sim
ulator config for scene/stage : data/scene_datasets/mp3d/B6ByNegPMKs/B6ByNegPMKs.glb and datas
et : default which is currently active dataset.

Should I blacklist it for the SAVi continuous environment ? Or before doing that, is there a way to force the loading of a specific scene, so that I can check if it really that scene which is causing a problem ?

Thanks you for your time.

@dosssman Hi,I see that you almost succeeded in using ss2.0 for SAVi,congratulations. I recently read the SAVi paper and found that the training of Descriptor Network may need to use binaural_rirs which is too big for my computer. Did you just used the on-policy training method to train the category? Or is there another way?

@zhibinQiu Hello again. Thanks. Not quite out of the woodwork yet :sweat_smile: though.

As long as you use the SS1.0 dataset based simulator for the SAVi task, the binaural_rirs dataset is required. Even when using the provided pre-trained weights for the category predictor, training the agent itself still requires the binaural_rirs dataset.

For the experimentation with the continuous simulator for SAVi, I use a very basic PPO implementation for now, that allows interaction with the environment. I did not explore the more sophisticated agent architecture that incorporates category and location predictors, etc...

If you want to avoid using the dataset, then continuous simulator is probably the only way. However, the SS2.0 SAVi simulator is also quite demanding on resources and slower than the audio navigation task, as well as it's dataset-based counterpart (SS1.0 SAVi).

Hi @dosssman I'm sorry for the terribly late replies these days ... work has been pretty hectic ...

It's pretty simple to check if you could load one environment, see this example on how to load one single environment: https://github.com/facebookresearch/habitat-sim/blob/main/examples/tutorials/audio_agent.py

I agree with you. The current issue I see using SAVi with SS 2 is that it might take forever to train the policy unless you have lots of GPUs. One hack in this case might be to compare the training curve with SAVi baseline within certain amount of training steps instead of waiting for the training curve to converge completely.

@ChanganVR No worries :sweat_smile: I did not have much time or compute to play around with Continuous SAVi recently.

Thanks for the tip. I will follow the example to check if that specific scene is causing the problem. Keep you posted on this issue.

Hello again.

After leaving the script running for around 8 hours, it ended up crashing with a segmentation fault, so I am not sure what to think about that:

lCoherence:1, meshSimplification:0, dumpWaveFiles:0, enableMaterials:1, writeIrToFile:0
[01:19:39:583015]:[Sensor] AudioSensor.cpp(50)::AudioSensor : [Audio]  Channel Layout :  channelType:3, channelCount:2
[01:19:39:583023]:[Sensor] AudioSensor.cpp(55)::AudioSensor : [Audio]  output directory not provided, will use /home/AudioSimulation
[01:19:39:583033]:[Sensor] AudioSensor.cpp(60)::AudioSensor : [Audio]  OutputDirectory :  /home/AudioSimulation
[01:19:39:583093]:[Sensor] AudioSensor.cpp(169)::setAudioMaterialsJSON : Set audio materials database to json file :  data/mp3d_material_config.json
[01:19:39:725056]:[Sensor] AudioSensor.cpp(81)::setAudioSourceTransform : [Audio]  Setting the audio source position :  Vector(50.0362, 1.60707, 18.1695) ]
[01:19:39:974825]:[Sensor] AudioSensor.cpp(90)::setAudioListenerTransform : [Audio]  Setting the agent transform : position [ Vector(16.6656, 1.5802, 23.0604) ], rotQuat[ Vector(1, 0, 2.45858e-05, 0) ]
[01:19:39:974889]:[Sensor] AudioSensor.cpp(278)::createAudioSimulator : [Audio]  Create audio simulator iteration:  0
[01:19:39:975326]:[Sensor] AudioSensor.cpp(114)::runSimulation : [Audio]  Running the audio simulator
[01:19:39:975344]:[Sensor] AudioSensor.cpp(119)::runSimulation : [Audio]  New initialization, will upload geometry and add the source at position :  Vector(50.0362, 1.60707, 18.1695)
[01:19:39:975359]:[Sensor] AudioSensor.cpp(126)::runSimulation : [Audio]  Loading semantic scene
[01:19:39:975368]:[Sensor] AudioSensor.cpp(293)::loadSemanticMesh : [Audio]  Loading semantic mesh
[01:19:46:497743]:[Sensor] AudioSensor.cpp(398)::loadSemanticMesh : [Audio]  Vertex count :  14069477 , Index count :  18120 , Material :  blinds , LastUpdate :  false
[01:19:46:498108]:[Sensor] AudioSensor.cpp(398)::loadSemanticMesh : [Audio]  Vertex count :  14069477 , Index count :  38226 , Material :  beam , LastUpdate :  false
[01:19:46:498494]:[Sensor] AudioSensor.cpp(398)::loadSemanticMesh : [Audio]  Vertex count :  14069477 , Index count :  232560 , Material :  tv_monitor , LastUpdate :  false
[01:19:46:499919]:[Sensor] AudioSensor.cpp(398)::loadSemanticMesh : [Audio]  Vertex count :  14069477 , Index count :  7590 , Material :  clothes , LastUpdate :  false
[01:19:46:501590]:[Sensor] AudioSensor.cpp(398)::loadSemanticMesh : [Audio]  Vertex count :  14069477 , Index count :  2014416 , Material :  lighting , LastUpdate :  false
[01:19:46:512326]:[Sensor] AudioSensor.cpp(398)::loadSemanticMesh : [Audio]  Vertex count :  14069477 , Index count :  2701404 , Material :  door , LastUpdate :  false
[01:19:46:556191]:[Sensor] AudioSensor.cpp(398)::loadSemanticMesh : [Audio]  Vertex count :  14069477 , Index count :  16538367 , Material :  misc , LastUpdate :  false
[01:19:46:718061]:[Sensor] AudioSensor.cpp(398)::loadSemanticMesh : [Audio]  Vertex count :  14069477 , Index count :  968469 , Material :  objects , LastUpdate :  false
[01:19:46:744479]:[Sensor] AudioSensor.cpp(398)::loadSemanticMesh : [Audio]  Vertex count :  14069477 , Index count :  11403030 , Material :  window , LastUpdate :  false
[01:19:46:866354]:[Sensor] AudioSensor.cpp(398)::loadSemanticMesh : [Audio]  Vertex count :  14069477 , Index count :  235524 , Material :  cabinet , LastUpdate :  false
[01:19:46:867226]:[Sensor] AudioSensor.cpp(398)::loadSemanticMesh : [Audio]  Vertex count :  14069477 , Index count :  1644 , Material :  sink , LastUpdate :  false
[01:19:46:867648]:[Sensor] AudioSensor.cpp(398)::loadSemanticMesh : [Audio]  Vertex count :  14069477 , Index count :  344244 , Material :  curtain , LastUpdate :  false
[01:19:46:870479]:[Sensor] AudioSensor.cpp(398)::loadSemanticMesh : [Audio]  Vertex count :  14069477 , Index count :  2240178 , Material :  chair , LastUpdate :  false
[01:19:46:876377]:[Sensor] AudioSensor.cpp(398)::loadSemanticMesh : [Audio]  Vertex count :  14069477 , Index count :  20472 , Material :  appliances , LastUpdate :  false
[01:19:46:878136]:[Sensor] AudioSensor.cpp(398)::loadSemanticMesh : [Audio]  Vertex count :  14069477 , Index count :  1925673 , Material :  table , LastUpdate :  false
[01:19:46:883357]:[Sensor] AudioSensor.cpp(398)::loadSemanticMesh : [Audio]  Vertex count :  14069477 , Index count :  43386 , Material :  furniture , LastUpdate :  false
[01:19:46:901332]:[Sensor] AudioSensor.cpp(398)::loadSemanticMesh : [Audio]  Vertex count :  14069477 , Index count :  7892847 , Material :  floor , LastUpdate :  false
[01:19:46:921961]:[Sensor] AudioSensor.cpp(398)::loadSemanticMesh : [Audio]  Vertex count :  14069477 , Index count :  107307 , Material :   , LastUpdate :  false
[01:19:46:945824]:[Sensor] AudioSensor.cpp(398)::loadSemanticMesh : [Audio]  Vertex count :  14069477 , Index count :  14172090 , Material :  ceiling , LastUpdate :  false
[01:19:46:983071]:[Sensor] AudioSensor.cpp(398)::loadSemanticMesh : [Audio]  Vertex count :  14069477 , Index count :  3783 , Material :  board_panel , LastUpdate :  false
[01:19:46:983246]:[Sensor] AudioSensor.cpp(398)::loadSemanticMesh : [Audio]  Vertex count :  14069477 , Index count :  96495 , Material :  counter , LastUpdate :  false
[01:19:46:984028]:[Sensor] AudioSensor.cpp(398)::loadSemanticMesh : [Audio]  Vertex count :  14069477 , Index count :  409017 , Material :  seating , LastUpdate :  false
[01:19:46:985439]:[Sensor] AudioSensor.cpp(398)::loadSemanticMesh : [Audio]  Vertex count :  14069477 , Index count :  100233 , Material :  void , LastUpdate :  false
[01:19:46:985807]:[Sensor] AudioSensor.cpp(398)::loadSemanticMesh : [Audio]  Vertex count :  14069477 , Index count :  43341 , Material :  sofa , LastUpdate :  false
[01:19:46:988849]:[Sensor] AudioSensor.cpp(398)::loadSemanticMesh : [Audio]  Vertex count :  14069477 , Index count :  1171878 , Material :  column , LastUpdate :  false
[01:19:46:992219]:[Sensor] AudioSensor.cpp(398)::loadSemanticMesh : [Audio]  Vertex count :  14069477 , Index count :  23268 , Material :  mirror , LastUpdate :  false
[01:19:46:992454]:[Sensor] AudioSensor.cpp(398)::loadSemanticMesh : [Audio]  Vertex count :  14069477 , Index count :  128607 , Material :  plant , LastUpdate :  false
[01:19:47:026144]:[Sensor] AudioSensor.cpp(398)::loadSemanticMesh : [Audio]  Vertex count :  14069477 , Index count :  14763747 , Material :  wall , LastUpdate :  false
[01:19:47:321226]:[Sensor] AudioSensor.cpp(398)::loadSemanticMesh : [Audio]  Vertex count :  14069477 , Index count :  65307 , Material :  picture , LastUpdate :  false
[01:19:47:321820]:[Sensor] AudioSensor.cpp(398)::loadSemanticMesh : [Audio]  Vertex count :  14069477 , Index count :  150723 , Material :  shelving , LastUpdate :  false
[01:19:47:322693]:[Sensor] AudioSensor.cpp(398)::loadSemanticMesh : [Audio]  Vertex count :  14069477 , Index count :  273552 , Material :  chest_of_drawers , LastUpdate :  false
[01:19:47:323733]:[Sensor] AudioSensor.cpp(398)::loadSemanticMesh : [Audio]  Vertex count :  14069477 , Index count :  46209 , Material :  stool , LastUpdate :  false
[01:19:47:323926]:[Sensor] AudioSensor.cpp(398)::loadSemanticMesh : [Audio]  Vertex count :  14069477 , Index count :  714 , Material :  towel , LastUpdate :  true
[01:20:22:700295]:[Sensor] AudioSensor.cpp(143)::runSimulation : [Audio]  Adding source at position :  Vector(50.0362, 1.60707, 18.1695)
[01:20:22:702291]:[Sensor] AudioSensor.cpp(151)::runSimulation : Running simulation, folder :  /home/AudioSimulation0
[01:20:22:714484]:[Sensor] AudioSensor.cpp(259)::getObservationSpace : [Audio]  getObservationSpace -> [ChannelCount] :  2 , [SampleCount] :  4740
[01:20:23:137666]:[Sensor] AudioSensor.cpp(90)::setAudioListenerTransform : [Audio]  Setting the agent transform : position [ Vector(68.0362, 1.48715, 20.1695) ], rotQuat[ Vector(0.707107, 0, 0.707107, 0) ]
[01:20:23:137734]:[Sensor] AudioSensor.cpp(278)::createAudioSimulator : [Audio]  Create audio simulator iteration:  1
[01:20:23:137753]:[Sensor] AudioSensor.cpp(114)::runSimulation : [Audio]  Running the audio simulator
[01:20:23:137764]:[Sensor] AudioSensor.cpp(151)::runSimulation : Running simulation, folder :  /home/AudioSimulation1
[01:20:23:140617]:[Sensor] AudioSensor.cpp(259)::getObservationSpace : [Audio]  getObservationSpace -> [ChannelCount] :  2 , [SampleCount] :  9035
./runs.sh: line 62: 297162 Segmentation fault      (core dumped) python ppo_savi.py --exp-name "ppo_savi_continuous" --total-steps 10000000 --wandb --wandb-project ss-hab --wandb-entity dosssman --logdir-prefix $LOGDIR_PREFIX --seed $seed

From the output above, I guess the continuous simulator is indeed working, but not stable enough.

Hi @dosssman @ChanganVR, I got the same problem about 3 hours after I start training with following command python ss_baselines/av_nav/run.py --exp-config ss_baselines/av_nav/config/audionav/mp3d/train_telephone/audiogoal_depth_ddppo.yaml --model-dir data/models/ss2/mp3d/dav_nav CONTINUOUS True

I build the environment following the step mentioned in #80, with the different is that my GPUs are two RTX 3080 with 10G memory, and I use cuda-11.6 installed from NVIDIA's website, cudatoolkit=11.6 and python3.7 installed by conda.

I built the habitat-sim with --headless to use my two GPUs although I have a display attached to my computer (though I didn't change the audiogoal_depth_ddppo.yaml so the process just run on my first GPU with index 0

After I got this error, I rerun the command with USE_SYNC_VECENV True and got the following result:


Thread 0x00007f6000a8c700 (most recent call first):

Thread 0x00007f600128d700 (most recent call first):

Thread 0x00007f6001a8e700 (most recent call first):
  File "/home/shizi9/miniconda3/envs/habitat/lib/python3.7/threading.py", line 300 in wait
  File "/home/shizi9/miniconda3/envs/habitat/lib/python3.7/queue.py", line 179 in get
  File "/home/shizi9/miniconda3/envs/habitat/lib/python3.7/site-packages/tensorboard/summary/writer/event_file_writer.py", line 227 in run
  File "/home/shizi9/miniconda3/envs/habitat/lib/python3.7/threading.py", line 926 in _bootstrap_inner
  File "/home/shizi9/miniconda3/envs/habitat/lib/python3.7/threading.py", line 890 in _bootstrap

Current thread 0x00007f656146a340 (most recent call first):
  File "/home/shizi9/miniconda3/envs/habitat/lib/python3.7/site-packages/habitat_sim-0.2.2-py3.7-linux-x86_64.egg/habitat_sim/simulator.py", line 661 in draw_observation
  File "/home/shizi9/miniconda3/envs/habitat/lib/python3.7/site-packages/habitat_sim-0.2.2-py3.7-linux-x86_64.egg/habitat_sim/simulator.py", line 436 in get_sensor_observations
  File "/home/shizi9/miniconda3/envs/habitat/lib/python3.7/site-packages/habitat_sim-0.2.2-py3.7-linux-x86_64.egg/habitat_sim/simulator.py", line 506 in step
  File "/home/shizi9/Workspace/sound-spaces/soundspaces/continuous_simulator.py", line 391 in step
  File "/home/shizi9/Workspace/habitat-lab/habitat/tasks/nav/nav.py", line 1005 in step
  File "/home/shizi9/Workspace/habitat-lab/habitat/core/embodied_task.py", line 303 in step
  File "/home/shizi9/Workspace/habitat-lab/habitat/core/env.py", line 267 in step
  File "/home/shizi9/Workspace/habitat-lab/habitat/core/env.py", line 410 in step
  File "/home/shizi9/miniconda3/envs/habitat/lib/python3.7/contextlib.py", line 74 in inner
  File "/home/shizi9/Workspace/sound-spaces/ss_baselines/common/environments.py", line 66 in step
  File "/home/shizi9/Workspace/sound-spaces/ss_baselines/common/sync_vector_env.py", line 82 in __call__
  File "/home/shizi9/Workspace/sound-spaces/ss_baselines/common/sync_vector_env.py", line 409 in <listcomp>
  File "/home/shizi9/Workspace/sound-spaces/ss_baselines/common/sync_vector_env.py", line 409 in step
  File "/home/shizi9/Workspace/sound-spaces/ss_baselines/av_nav/ddppo/ppo_trainer.py", line 243 in _collect_rollout_step
  File "/home/shizi9/Workspace/sound-spaces/ss_baselines/av_nav/ddppo/ddppo_trainer.py", line 283 in train
  File "ss_baselines/av_nav/run.py", line 95 in main
  File "ss_baselines/av_nav/run.py", line 101 in <module>
Aborted (core dumped)

At first I think it's the problem with my GPU memory, so I check the GPU usage with nvidia-smi, but it seems allright, so I check the log along with the nvidia-smi log and find the following log:

...
[13:16:30:828247]:[Physics] PhysicsManager.cpp(50)::~PhysicsManager : Deconstructing PhysicsManager
[13:16:30:828391]:[Scene] SceneManager.h(25)::~SceneManager : Deconstructing SceneManager
...
[13:16:30:996949]:[Metadata] MetadataMediator.cpp(66)::setSimulatorConfiguration : Set new simulator config for scene/stage : data/scene_datasets/mp3d/aayBHfsNo7d/aayBHfsNo7d.glb and dataset : default which is currently active dataset.
Renderer: NVIDIA GeForce RTX 3080/PCIe/SSE2 by NVIDIA Corporation
OpenGL version: 4.6.0 NVIDIA 510.85.02
Using optional features:
    GL_ARB_vertex_array_object
    GL_ARB_ES2_compatibility
    GL_ARB_separate_shader_objects
    GL_ARB_robustness
    GL_ARB_texture_storage
    GL_ARB_invalidate_subdata
    GL_ARB_texture_storage_multisample
    GL_ARB_multi_bind
    GL_ARB_direct_state_access
    GL_ARB_get_texture_sub_image
    GL_ARB_texture_filter_anisotropic
    GL_KHR_debug
Using driver workarounds:
    no-forward-compatible-core-context
    nv-egl-incorrect-gl11-function-pointers
    no-layout-qualifiers-on-old-glsl
    nv-zero-context-profile-mask
    nv-implementation-color-read-format-dsa-broken
    nv-cubemap-inconsistent-compressed-image-size
    nv-cubemap-broken-full-compressed-image-query
    nv-compressed-block-size-in-bits
[13:16:31:011477]:[Metadata] MetadataMediator.cpp(297)::getSceneInstanceAttributesByName : Dataset : default has no preloaded SceneInstanceAttributes or StageAttributes named : data/scene_datasets/mp3d/aayBHfsNo7d/aayBHfsNo7d.glb so loading/creating a new StageAttributes with this name, and then creating a SceneInstanceAttributes with the same name that references this stage.
...
[13:16:37:981274]:[Sensor] AudioSensor.cpp(398)::loadSemanticMesh : [Audio]  Vertex count :  3095346 , Index count :  87564 , Material :  toilet , LastUpdate :  false
[13:16:37:981484]:[Sensor] AudioSensor.cpp(398)::loadSemanticMesh : [Audio]  Vertex count :  3095346 , Index count :  22122 , Material :  towel , LastUpdate :  true
[13:16:42:593850]:[Sensor] AudioSensor.cpp(143)::runSimulation : [Audio]  Adding source at position :  Vector(7.78595, 5.06549, 5.18635)
[13:16:42:594712]:[Sensor] AudioSensor.cpp(151)::runSimulation : Running simulation, folder :  /home/AudioSimulation0
[13:16:42:599732]:[Sensor] AudioSensor.cpp(259)::getObservationSpace : [Audio]  getObservationSpace -> [ChannelCount] :  2 , [SampleCount] :  1120
[13:16:42:601085]:[Sensor] AudioSensor.cpp(90)::setAudioListenerTransform : [Audio]  Setting the agent transform : position [ Vector(4.78595, 4.96549, 4.18635) ], rotQuat[ Vector(-0.707107, 0, 0.707107, 0) ]
[13:16:42:601127]:[Sensor] AudioSensor.cpp(278)::createAudioSimulator : [Audio]  Create audio simulator iteration:  1
[13:16:42:601139]:[Sensor] AudioSensor.cpp(114)::runSimulation : [Audio]  Running the audio simulator
[13:16:42:601146]:[Sensor] AudioSensor.cpp(151)::runSimulation : Running simulation, folder :  /home/AudioSimulation1
[13:16:42:602625]:[Sensor] AudioSensor.cpp(259)::getObservationSpace : [Audio]  getObservationSpace -> [ChannelCount] :  2 , [SampleCount] :  2825

It seems that the previous scene was destoryed and the process will load a new scene for training. However, different from the previous success taring loop, it start a new rendering subprocess, like Renderer: NVIDIA GeForce RTX 3080/PCIe/SSE2 by NVIDIA Corporation right? But I see the memory usage of my GPU dropped from about 4.5G to 2.8G, and according to the success training loop, the step like runSimulation and getObservationSpace will run sequencelly for all subprocess setted in config NUM_PROCESSES. The reality is the process ended at setAudioListenerTransform for the previous scene. So I gusess there are somethings wrong with the start of the new NVIDIA rendering process. And I will rerun the command with NUM_PROCESSES=4 to prevent the GPU memory run out

I will attach the log and the nvidia-smi log files, the above question happed at about 13:16:31 in the log files, and could you please give me some suggestions?

And thanks a lot for this work @ChanganVR and the installation instruction @dosssman ddppo_log.zip gpu_usage.zip

@dosssman That's weird. I've encoutered segmentation fault in the past but it always happens at the beginning due to loading incorrect material files etc.

The only thing I could possibly think of is the simulator was loading one bad environment and for Matterport3D, I had to exlcude this environment:

https://github.com/facebookresearch/sound-spaces/blob/a1f882f248da13bd9ef51c42cb0fe40a44f4e1d0/ss_baselines/common/env_utils.py#L46-L48

Could you check the longer log and see which environment was loaded?

Any chance you could try to find the latest scene it loads for your before crashing ? If it is the same as mine, then there might be some hope by adding it to the blacklist. Also, is there anyway for you to track the system memory in usage ?

In the meantime, if you are hurried to get some results, I would recommend first trying with the dataset SS 1.0 version instead, which runs order of magnitude faster. Once you have confirmed your algorithm is working as intended, you might try with the continuous SS2.0 version.

Best of luck.

@dosssman That's weird. I've encoutered segmentation fault in the past but it always happens at the beginning due to loading incorrect material files etc. The only thing I could possibly think of is the simulator was loading one bad environment and for Matterport3D, I had to exlcude this environment: https://github.com/facebookresearch/sound-spaces/blob/a1f882f248da13bd9ef51c42cb0fe40a44f4e1d0/ss_baselines/common/env_utils.py#L46-L48

Could you check the longer log and see which environment was loaded?

Any chance you could try to find the latest scene it loads for your before crashing ? If it is the same as mine, then there might be some hope by adding it to the blacklist. Also, is there anyway for you to track the system memory in usage ?

In the meantime, if you are hurried to get some results, I would recommend first trying with the dataset SS 1.0 version instead, which runs order of magnitude faster. Once you have confirmed your algorithm is working as intended, you might try with the continuous SS2.0 version.

Best of luck.

I try to add the scene into the blacklist but it still broke with the next scene, so I don't think it's the dataset's fault

And I have successed in SS1.0 with the training example in the readme of SoundSpace, the training process finished but the validation not finish because I change the NUM_UPDATES so the script may try to find the following check point. I will try it latter.

As for the system memory, because I encountered the OOM error so I add a 2T swap with my 128G memory, and I check it with free -h occasionally, so I don't think it's the memory problem.

I will keep trying, thanks a lot.

I made some mistakes, the python log show that the process chocked at wait function (maybe on the parent process) and draw_observation function (maybe on the subprocess), so this error may occur on the new scene's getObservationSpace rather than the old scene's setAudioListenerTransform

I also found that the GPU memory usage increased at loadStage so the GPU usage dropping on the ~PhysicsManager and so on mentioned above should be normal. And some "bad scene" (bad environment) interrupted at one training successed on another training process because every training process varied owing to the multiple subprocesses running parallelly. So it should not the bad environment's fault.

Maybe I should go deeper into the code to find some solutions

@shizi991016 sorry for the delay. Regarding your question, to double check whether the scene assets are ok, you could run this script for mp3d dataset and only render one iteration for each environment for quick debug. This will help you isolate the rendering from navigation/RL.

FYI, I also just udpated the code/API and you can find a quick tutorial here. It might be worth trying out as well.

If after trying the last two steps, you still have this issue, feel free to send me an email to schedule a meeting and I can help you troubleshoot the issue over a call.

@ChanganVR Thank you very much, I will try these two steps as soon

@ChanganVR Hi changan, I try these two steps. For the mp3d dataset rendering, I use python PanoIR/render_panoIR.py --dataset mp3d --num-per-scene 1 to form the data and I got 84 directories with scene id under data/PanoIR, with no error in the log file. I think the mp3d dataset is ok.

For the tutorial, I got the following error: AttributeError: 'habitat_sim._ext.habitat_sim_bindings.RLRAudioPropagationChannelLayout' object has no attribute 'type' Does this error matter? the full log is attached at the end

I rebuilt the habitat-sim from the commit with checkout 80f8e31 via source, with command python setup.py install --headless --with-cuda --bullet --audio --cmake, and I don't know how to deal with the error

So could you please give me some suggestions? Thank you very much

BTW, I found in the /PanoIR/README.md, Reproduce section, the command python PanoIR/read_panoIR.py --dataset mp3d maybe python PanoIR/render_panoIR.py --dataset mp3d?

The full log is here:

[12:29:44:577255]:[Metadata] AttributesManagerBase.h(380)::createFromJsonOrDefaultInternal : <Dataset>: Proposing JSON name : default.scene_dataset_config.json from original name : default| This file does not exist.
[12:29:44:577343]:[Metadata] AssetAttributesManager.cpp(121)::createObject : Asset attributes (capsule3DSolid:capsule3DSolid_hemiRings_4_cylRings_1_segments_12_halfLen_0.75_useTexCoords_false_useTangents_false) created and registered.
[12:29:44:577371]:[Metadata] AssetAttributesManager.cpp(121)::createObject : Asset attributes (capsule3DWireframe:capsule3DWireframe_hemiRings_8_cylRings_1_segments_16_halfLen_1) created and registered.
[12:29:44:577388]:[Metadata] AssetAttributesManager.cpp(121)::createObject : Asset attributes (coneSolid:coneSolid_segments_12_halfLen_1.25_rings_1_useTexCoords_false_useTangents_false_capEnd_true) created and registered.
[12:29:44:577402]:[Metadata] AssetAttributesManager.cpp(121)::createObject : Asset attributes (coneWireframe:coneWireframe_segments_32_halfLen_1.25) created and registered.
[12:29:44:577414]:[Metadata] AssetAttributesManager.cpp(121)::createObject : Asset attributes (cubeSolid:cubeSolid) created and registered.
[12:29:44:577424]:[Metadata
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
/tmp/ipykernel_175045/2581157944.py in <module>
     33 audio_sensor_spec.uuid = "audio_sensor"
     34 audio_sensor_spec.enableMaterials = True
---> 35 audio_sensor_spec.channelLayout.type = habitat_sim.sensor.RLRAudioPropagationChannelLayoutType.Binaural
     36 audio_sensor_spec.channelLayout.channelCount = 1
     37 # audio sensor location set with respect to the agent

AttributeError: 'habitat_sim._ext.habitat_sim_bindings.RLRAudioPropagationChannelLayout' object has no attribute 'type'

] AssetAttributesManager.cpp(121)::createObject : Asset attributes (cubeWireframe:cubeWireframe) created and registered.
[12:29:44:577439]:[Metadata] AssetAttributesManager.cpp(121)::createObject : Asset attributes (cylinderSolid:cylinderSolid_rings_1_segments_12_halfLen_1_useTexCoords_false_useTangents_false_capEnds_true) created and registered.
[12:29:44:577453]:[Metadata] AssetAttributesManager.cpp(121)::createObject : Asset attributes (cylinderWireframe:cylinderWireframe_rings_1_segments_32_halfLen_1) created and registered.
[12:29:44:577465]:[Metadata] AssetAttributesManager.cpp(121)::createObject : Asset attributes (icosphereSolid:icosphereSolid_subdivs_1) created and registered.
[12:29:44:577477]:[Metadata] AssetAttributesManager.cpp(121)::createObject : Asset attributes (icosphereWireframe:icosphereWireframe_subdivs_1) created and registered.
[12:29:44:577491]:[Metadata] AssetAttributesManager.cpp(121)::createObject : Asset attributes (uvSphereSolid:uvSphereSolid_rings_8_segments_16_useTexCoords_false_useTangents_false) created and registered.
[12:29:44:577503]:[Metadata] AssetAttributesManager.cpp(121)::createObject : Asset attributes (uvSphereWireframe:uvSphereWireframe_rings_16_segments_32) created and registered.
[12:29:44:577511]:[Metadata] AssetAttributesManager.cpp(110)::AssetAttributesManager : Built default primitive asset templates : 12
[12:29:44:577688]:[Metadata] SceneDatasetAttributesManager.cpp(35)::createObject : File (default) not found, so new default dataset attributes created  and registered.
[12:29:44:577695]:[Metadata] MetadataMediator.cpp(120)::createSceneDataset : Dataset default successfully created.
[12:29:44:577707]:[Metadata] AttributesManagerBase.h(380)::createFromJsonOrDefaultInternal : <Physics Manager>: Proposing JSON name : ./data/default.physics_config.json from original name : ./data/default.physics_config.json| This file exists.
[12:29:44:577794]:[Metadata] PhysicsAttributesManager.cpp(26)::createObject : JSON Configuration File (./data/default.physics_config.json) based physics manager attributes created and registered.
[12:29:44:577801]:[Metadata] MetadataMediator.cpp(210)::setActiveSceneDatasetName : Attempting to create new dataset data/scene_datasets/mp3d/mp3d.scene_dataset_config.json
[12:29:44:577809]:[Metadata] AttributesManagerBase.h(380)::createFromJsonOrDefaultInternal : <Dataset>: Proposing JSON name : data/scene_datasets/mp3d/mp3d.scene_dataset_config.json from original name : data/scene_datasets/mp3d/mp3d.scene_dataset_config.json| This file does not exist.
[12:29:44:577826]:[Metadata] AssetAttributesManager.cpp(121)::createObject : Asset attributes (capsule3DSolid:capsule3DSolid_hemiRings_4_cylRings_1_segments_12_halfLen_0.75_useTexCoords_false_useTangents_false) created and registered.
[12:29:44:577841]:[Metadata] AssetAttributesManager.cpp(121)::createObject : Asset attributes (capsule3DWireframe:capsule3DWireframe_hemiRings_8_cylRings_1_segments_16_halfLen_1) created and registered.
[12:29:44:577855]:[Metadata] AssetAttributesManager.cpp(121)::createObject : Asset attributes (coneSolid:coneSolid_segments_12_halfLen_1.25_rings_1_useTexCoords_false_useTangents_false_capEnd_true) created and registered.
[12:29:44:577869]:[Metadata] AssetAttributesManager.cpp(121)::createObject : Asset attributes (coneWireframe:coneWireframe_segments_32_halfLen_1.25) created and registered.
[12:29:44:577880]:[Metadata] AssetAttributesManager.cpp(121)::createObject : Asset attributes (cubeSolid:cubeSolid) created and registered.
[12:29:44:577889]:[Metadata] AssetAttributesManager.cpp(121)::createObject : Asset attributes (cubeWireframe:cubeWireframe) created and registered.
[12:29:44:577902]:[Metadata] AssetAttributesManager.cpp(121)::createObject : Asset attributes (cylinderSolid:cylinderSolid_rings_1_segments_12_halfLen_1_useTexCoords_false_useTangents_false_capEnds_true) created and registered.
[12:29:44:577914]:[Metadata] AssetAttributesManager.cpp(121)::createObject : Asset attributes (cylinderWireframe:cylinderWireframe_rings_1_segments_32_halfLen_1) created and registered.
[12:29:44:577925]:[Metadata] AssetAttributesManager.cpp(121)::createObject : Asset attributes (icosphereSolid:icosphereSolid_subdivs_1) created and registered.
[12:29:44:577936]:[Metadata] AssetAttributesManager.cpp(121)::createObject : Asset attributes (icosphereWireframe:icosphereWireframe_subdivs_1) created and registered.
[12:29:44:577948]:[Metadata] AssetAttributesManager.cpp(121)::createObject : Asset attributes (uvSphereSolid:uvSphereSolid_rings_8_segments_16_useTexCoords_false_useTangents_false) created and registered.
[12:29:44:577959]:[Metadata] AssetAttributesManager.cpp(121)::createObject : Asset attributes (uvSphereWireframe:uvSphereWireframe_rings_16_segments_32) created and registered.
[12:29:44:577966]:[Metadata] AssetAttributesManager.cpp(110)::AssetAttributesManager : Built default primitive asset templates : 12
[12:29:44:578120]:[Metadata] SceneDatasetAttributesManager.cpp(35)::createObject : File (data/scene_datasets/mp3d/mp3d.scene_dataset_config.json) not found, so new default dataset attributes created  and registered.
[12:29:44:578126]:[Metadata] MetadataMediator.cpp(120)::createSceneDataset : Dataset data/scene_datasets/mp3d/mp3d.scene_dataset_config.json successfully created.
[12:29:44:578132]:[Metadata] MetadataMediator.cpp(217)::setActiveSceneDatasetName : Attempt to create new dataset data/scene_datasets/mp3d/mp3d.scene_dataset_config.json   succeeded. Currently active dataset : data/scene_datasets/mp3d/mp3d.scene_dataset_config.json
[12:29:44:578137]:[Metadata] MetadataMediator.cpp(175)::setCurrPhysicsAttributesHandle : Old physics manager attributes  changed to ./data/default.physics_config.json successfully.
[12:29:44:578144]:[Metadata] MetadataMediator.cpp(66)::setSimulatorConfiguration : Set new simulator config for scene/stage : data/scene_datasets/mp3d/UwV83HsGsw3/UwV83HsGsw3.glb and dataset : data/scene_datasets/mp3d/mp3d.scene_dataset_config.json which is currently active dataset.
[12:29:44:578435]:[Metadata] MetadataMediator.cpp(297)::getSceneInstanceAttributesByName : Dataset : data/scene_datasets/mp3d/mp3d.scene_dataset_config.json has no preloaded SceneInstanceAttributes or StageAttributes named : data/scene_datasets/mp3d/UwV83HsGsw3/UwV83HsGsw3.glb so loading/creating a new StageAttributes with this name, and then creating a SceneInstanceAttributes with the same name that references this stage.
[12:29:44:578446]:[Metadata] AttributesManagerBase.h(380)::createFromJsonOrDefaultInternal : <Stage Template>: Proposing JSON name : data/scene_datasets/mp3d/UwV83HsGsw3/UwV83HsGsw3.stage_config.json from original name : data/scene_datasets/mp3d/UwV83HsGsw3/UwV83HsGsw3.glb| This file does not exist.
[12:29:44:578492]:[Metadata] AbstractObjectAttributesManagerBase.h(184)::createObject : File (data/scene_datasets/mp3d/UwV83HsGsw3/UwV83HsGsw3.glb) exists but is not a recognized config filename extension, so new default Stage Template attributes created and registered.
[12:29:44:578526]:[Metadata] SceneDatasetAttributes.cpp(47)::addNewSceneInstanceToDataset : Dataset : 'mp3d' : Stage Attributes 'data/scene_datasets/mp3d/UwV83HsGsw3/UwV83HsGsw3.glb' specified in Scene Attributes exists in dataset library.
[12:29:44:578534]:[Metadata] SceneDatasetAttributes.cpp(84)::addNewSceneInstanceToDataset : Dataset : 'mp3d' : Lighting Layout Attributes 'no_lights' specified in Scene Attributes but does not exist in dataset, so creating.
[12:29:44:578542]:[Metadata] AttributesManagerBase.h(380)::createFromJsonOrDefaultInternal : <Lighting Layout>: Proposing JSON name : no_lights.lighting_config.json from original name : no_lights| This file does not exist.
[12:29:44:578555]:[Metadata] LightLayoutAttributesManager.cpp(36)::createObject : File (no_lights) not found, so new default light layout attributes created and registered.
[12:29:44:578564]:[Sim] Simulator.cpp(237)::createSceneInstance : Navmesh file location in scene instance : data/scene_datasets/mp3d/UwV83HsGsw3/UwV83HsGsw3.navmesh
[12:29:44:578571]:[Sim] Simulator.cpp(242)::createSceneInstance : Loading navmesh from data/scene_datasets/mp3d/UwV83HsGsw3/UwV83HsGsw3.navmesh
[12:29:44:578585]:[Nav] PathFinder.cpp(852)::loadNavMesh : NavMeshSettings aren't present, guessing that they are the default
[12:29:44:578849]:[Sim] Simulator.cpp(244)::createSceneInstance : Navmesh Loaded.
[12:29:44:578861]:[Scene] SceneGraph.h(85)::createDrawableGroup : Created DrawableGroup: 
[12:29:44:578867]:[Assets] ResourceManager.cpp(297)::loadSemanticSceneDescriptor : SceneInstance : data/scene_datasets/mp3d/UwV83HsGsw3/UwV83HsGsw3.glb proposed Semantic Scene Descriptor filename : data/scene_datasets/mp3d/UwV83HsGsw3/UwV83HsGsw3.house
[12:29:44:625109]:[Assets] ResourceManager.cpp(308)::loadSemanticSceneDescriptor : SSD with SceneInstanceAttributes-provided name  data/scene_datasets/mp3d/UwV83HsGsw3/UwV83HsGsw3.house successfully found and loaded
[12:29:44:625166]:[Sim] Simulator.cpp(302)::createSceneInstance : Using scene instance-specified Light key : - no_lights -
[12:29:44:625176]:[Metadata] MetadataMediator.cpp(66)::setSimulatorConfiguration : Set new simulator config for scene/stage : data/scene_datasets/mp3d/UwV83HsGsw3/UwV83HsGsw3.glb and dataset : data/scene_datasets/mp3d/mp3d.scene_dataset_config.json which is currently active dataset.
[12:29:44:625200]:[Sim] Simulator.cpp(412)::instanceStageForSceneAttributes : Start to load stage named : data/scene_datasets/mp3d/UwV83HsGsw3/UwV83HsGsw3.glb with render asset : data/scene_datasets/mp3d/UwV83HsGsw3/UwV83HsGsw3.glb and collision asset : data/scene_datasets/mp3d/UwV83HsGsw3/UwV83HsGsw3.glb
[12:29:44:625232]:[Assets] ResourceManager.cpp(691)::createStageAssetInfosFromAttributes : Frame :{"up":[0,0,1],"front":[0,1,0],"origin":[0,0,0]} for render mesh named : data/scene_datasets/mp3d/UwV83HsGsw3/UwV83HsGsw3.glb|{"up":[0,0,1],"front":[0,1,0],"origin":[0,0,0]} for semantic mesh named : data/scene_datasets/mp3d/UwV83HsGsw3/UwV83HsGsw3_semantic.ply with type specified as semantic|Semantic Txtrs : False
[12:29:44:625247]:[Assets] ResourceManager.cpp(431)::loadStage : Loading Semantic Stage mesh : data/scene_datasets/mp3d/UwV83HsGsw3/UwV83HsGsw3_semantic.ply
[12:29:44:625253]:[Scene] SceneGraph.h(85)::createDrawableGroup : Created DrawableGroup: 
[12:29:44:625257]:[Assets] ResourceManager.cpp(973)::loadStageInternal : Attempting to load stage data/scene_datasets/mp3d/UwV83HsGsw3/UwV83HsGsw3_semantic.ply 
[12:29:44:625264]:[Assets] ResourceManager.cpp(852)::loadRenderAsset : Loading Semantic Mesh asset named: data/scene_datasets/mp3d/UwV83HsGsw3/UwV83HsGsw3_semantic.ply
[12:29:54:832452]:[Assets] ResourceManager.cpp(464)::loadStage : Semantic Stage mesh : data/scene_datasets/mp3d/UwV83HsGsw3/UwV83HsGsw3_semantic.ply loaded.
[12:29:54:832502]:[Assets] ResourceManager.cpp(507)::loadStage : Start load render asset data/scene_datasets/mp3d/UwV83HsGsw3/UwV83HsGsw3.glb .
[12:29:54:832506]:[Assets] ResourceManager.cpp(973)::loadStageInternal : Attempting to load stage data/scene_datasets/mp3d/UwV83HsGsw3/UwV83HsGsw3.glb 
[12:29:54:832523]:[Assets] ResourceManager.cpp(855)::loadRenderAsset : Loading general asset named: data/scene_datasets/mp3d/UwV83HsGsw3/UwV83HsGsw3.glb
[12:29:54:933448]:[Sim] Simulator.cpp(439)::instanceStageForSceneAttributes : Successfully loaded stage named : data/scene_datasets/mp3d/UwV83HsGsw3/UwV83HsGsw3.glb
[12:29:54:933549]:[Sim] Simulator.cpp(205)::reconfigure : CreateSceneInstance success == true for active scene name : data/scene_datasets/mp3d/UwV83HsGsw3/UwV83HsGsw3.glb  without renderer.
[12:29:54:934852]:[Nav] PathFinder.cpp(852)::loadNavMesh : NavMeshSettings aren't present, guessing that they are the default
[12:29:54:935196]:[Sensor] AudioSensor.cpp(24)::AudioSensorSpec : [Audio]  AudioSensorSpec constructor
[12:29:54:937862]:[Physics] PhysicsManager.cpp(50)::~PhysicsManager : Deconstructing PhysicsManager
[12:29:54:937900]:[Scene] SceneManager.h(25)::~SceneManager : Deconstructing SceneManager
[12:29:54:937905]:[Scene] SceneGraph.h(25)::~SceneGraph : Deconstructing SceneGraph
[12:29:54:937999]:[Scene] SceneGraph.h(25)::~SceneGraph : Deconstructing SceneGraph
[12:29:54:938236]:[Scene] SemanticScene.h(47)::~SemanticScene : Deconstructing SemanticScene
[12:29:54:941741]:[Sim] Simulator.cpp(66)::~Simulator : Deconstructing Simulator

@shizi991016 as instructed on the readme page, you'll also need to update the RLR-Audio-Propagation binaries to the latest version: https://github.com/facebookresearch/rlr-audio-propagation/tree/UpdateAPI

@ChanganVR Hi changan, I followed your instructions to rebuild the habitat-sim. I noticed the habitat-sim and RLR-Audio-Propagation version updated after your last commit of soundspace, so I use the version committed on 7 Sep. I got a similar result by running the python code, with log:

Sample source location:  [-1.0204104e+01 -9.4709396e-03  2.6049542e+00]
CreateContext: Context created
(2, 55469)
True
0.45320600271224976
362.2641509433962
48000 (233873,)
RT60 of the rendered IR is 0.2675 seconds

Figure_1 Figure_2 and the convolved_vocal.wav sounds good

I think the tutorial worked fine on my conda environment, so I try to train the continuous navigation agent using the same conda environment but got this error:

No module named 'habitat_sim.robots'
No module named 'habitat_sim.robots'
No module named 'habitat_sim.robots'
No module named 'habitat_sim.robots'
Traceback (most recent call last):
  File "/home/shizi9/miniconda3/envs/ss2/lib/python3.7/runpy.py", line 183, in _run_module_as_main
    mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
  File "/home/shizi9/miniconda3/envs/ss2/lib/python3.7/runpy.py", line 109, in _get_module_details
    __import__(pkg_name)
  File "/home/shizi9/Workspace/ss2/sound-spaces/ss_baselines/av_nav/__init__.py", line 9, in <module>
    from ss_baselines.av_nav.ppo.ppo_trainer import PPOTrainer
  File "/home/shizi9/Workspace/ss2/sound-spaces/ss_baselines/av_nav/ppo/ppo_trainer.py", line 24, in <module>
    from habitat import Config, logger
  File "/home/shizi9/Workspace/ss2/habitat-lab/habitat/__init__.py", line 9, in <module>
    from habitat.core.benchmark import Benchmark
  File "/home/shizi9/Workspace/ss2/habitat-lab/habitat/core/benchmark.py", line 18, in <module>
    from habitat.core.env import Env
  File "/home/shizi9/Workspace/ss2/habitat-lab/habitat/core/env.py", line 22, in <module>
    from habitat.tasks import make_task
  File "/home/shizi9/Workspace/ss2/habitat-lab/habitat/tasks/__init__.py", line 7, in <module>
    from habitat.tasks.registration import make_task  # noqa: F401
  File "/home/shizi9/Workspace/ss2/habitat-lab/habitat/tasks/registration.py", line 28, in <module>
    _try_register_rearrange_task()
  File "/home/shizi9/Workspace/ss2/habitat-lab/habitat/tasks/rearrange/__init__.py", line 33, in _try_register_rearrange_task
    import habitat.tasks.rearrange.actions
  File "/home/shizi9/Workspace/ss2/habitat-lab/habitat/tasks/rearrange/actions.py", line 15, in <module>
    from habitat.tasks.rearrange.rearrange_sim import RearrangeSim
  File "/home/shizi9/Workspace/ss2/habitat-lab/habitat/tasks/rearrange/rearrange_sim.py", line 33, in <module>
    from habitat_sim.robots import FetchRobot
ModuleNotFoundError: No module named 'habitat_sim.robots'

I never get this error before with habitat-sim 0.2.2(80f8e31). Maybe there are something wrong with this habitat_sim version?

And I had a closer look at the log of the tutorial, there are some log like:

[10:37:26:111310]:[Metadata] AttributesManagerBase.h(380)::createFromJsonOrDefaultInternal : <Dataset>: Proposing JSON name : default.scene_dataset_config.json from original name : default| This file does not exist.
...
[10:37:26:112821]:[Metadata] AttributesManagerBase.h(380)::createFromJsonOrDefaultInternal : <Stage Template>: Proposing JSON name : data/scene_datasets/mp3d/17DRP5sb8fy/17DRP5sb8fy.stage_config.json from original name : data/scene_datasets/mp3d/17DRP5sb8fy/17DRP5sb8fy.glb| This file does not exist.

Is this mean I loss some default configuration files?

I hope you can give me some suggestions. Thank you!

@shizi991016 I'm sorry that I missed your issue earlier. Have this issue been fixed? It might be due to some broken APIs, which should be easy to fix. If you haven't fixed it, I'll try to get to it in the next two days.

@ChanganVR Hi Changan, I was reading SS1.0 these days and didn't try to fix the bug of last commit. Because I only have two GPUs so I will focus on SS1.0 in the future. If I meet new problem of SS2.0 I will let you know as soon as possible.

@shizi991016 hey,

I faced the issue ModuleNotFoundError: No module named 'habitat_sim.robots' while playing around with SS20 and SS10.

I was able to resolve it by making sure that I used the right habitat-sim version for the corresponding simulator. i.e. For SS20, you will have to build habitat-sim using this branch and for SS10, use the latest stable habitat version. Remember to pip install -e . the soundspaces repo after building the habitat-sim repo.

I'd suggest that you use a separate conda environment for SS10 and SS20 as that works for me and the above error came as a result of building an incorrect version of habitat-sim and trying to use it with SS10.

Do let me know if this helps!

@sreeharshaparuchur1 Thank you very much! I will try this later

facebookresearch / sound-spaces

Using SAVi with SS2.0's continuous simulator ? #85