search

facebookresearch / sound-spaces

A first-of-its-kind acoustic simulation platform for audio-visual embodied AI research. It supports training and evaluating multiple tasks and applications.
https://soundspaces.org
Creative Commons Attribution 4.0 International
338 stars 55 forks source link

DDPPO Agent evaluation and video generation #80

Closed dosssman closed 2 years ago

dosssman commented 2 years ago

Dear @ChanganVR ,

Thank you very much for your continued support regarding Sound Spaces. It is quite a rare, but very refreshing experience. :bow: Thanks to your help, I have managed to get sound spaces 2.0 working, and training the provided DDPPO baseline.

Is there any way to get videos with audio logged into the tensorboard as the DDPPO agent trains, akin to how is done in the scripts/interactive_demo.py ?

Best clues I could find were the VIDEO_OPTION and VISUALIZATION_OPTION in the ss_baselines/av_nav/config/audionav/mp3d/train_telephone/audiogoal_depth_ddppo.yaml file. Namely, I respectively set them to "tensorboard", and "top_down_map" before passing it to the --exp-config flag, but it does not seem to generate anything along with the tensorboard logs.

While I am still digging through the code to find such features, I thought I might as well open an issue to verify if such features is supported.

I happened to find something that would match the use case in the ppo_trainer.py, but not sure how it could be ported to the ddppo_trianer.py.

Beside that, assuming I have a properly training DDPPO baseline, which saves files like checkpoint_150.pth, is there any scripts that could load those and generate trajectories with audio and video to evaluate the final agents

Looking forward to hear from you again.

Best regards.

dosssman commented 2 years ago

Although tangentially related to the original topic of this issue, do you happen to be familiar with this error when trying to execute python scripts/interactive_demon.yaml ?

Fatal Python error: (pygame parachute) Segmentation Fault

Current thread 0x00007f7967d06340 (most recent call first):
  File "/home/rousslan/anaconda3/envs/ss-hab-display/lib/python3.7/site-packages/habitat_sim-0.2.2-py3.7-linux-x86_64.egg/habitat_sim/simulator.py", line 661 in draw_observation
  File "/home/rousslan/anaconda3/envs/ss-hab-display/lib/python3.7/site-packages/habitat_sim-0.2.2-py3.7-linux-x86_64.egg/habitat_sim/simulator.py", line 436 in get_sensor_observations
  File "/home/rousslan/anaconda3/envs/ss-hab-display/lib/python3.7/site-packages/habitat_sim-0.2.2-py3.7-linux-x86_64.egg/habitat_sim/simulator.py", line 185 in reset
  File "/home/rousslan/random/rl/ss-hab-display/sound-spaces/soundspaces/continuous_simulator.py", line 364 in reset
  File "/home/rousslan/random/rl/ss-hab-display/habitat-lab/habitat/core/embodied_task.py", line 284 in reset
  File "/home/rousslan/random/rl/ss-hab-display/habitat-lab/habitat/core/env.py", line 252 in reset
  File "/home/rousslan/random/rl/ss-hab-display/habitat-lab/habitat/core/env.py", line 402 in reset
  File "/home/rousslan/anaconda3/envs/ss-hab-display/lib/python3.7/contextlib.py", line 74 in inner
  File "/home/rousslan/random/rl/ss-hab-display/sound-spaces/ss_baselines/common/environments.py", line 53 in reset
  File "scripts/interactive_demo.py", line 52 in interactive_demo
  File "scripts/interactive_demo.py", line 237 in main
  File "scripts/interactive_demo.py", line 244 in <module>
Aborted (core dumped)

I managed to get the interactive_demo to work when using an the internal graphics of the motherboard,. However, I have changed the rendering to use an NVIDIA RTX 3090 GPU instead, and since then, I have not been able to get the interactive_demo to work.

The traceback of the error is also not very information, so I would appreciate any insights on what might the cause, if there is no known fix already.

Thanks again for your time.

YHWmz commented 2 years ago

I also met the "Fatal Python error: (pygame parachute) Segmentation Fault" error when trying to run 

python scripts/interactive_demo.py

and

python ss_baselines/av_nav/run.py --exp-config ss_baselines/av_nav/config/audionav/replica/train_telephone/audiogoal_depth_ddppo.yaml --model-dir data/models/ss2/replica/dav_nav CONTINUOUS True

@dosssman Have you fixed this error? If so, could you tell me how to solve it?

dosssman commented 2 years ago

Helli there.

I only get that error for the interactive mode, and since it is not really needed for RL agent training anyway, I kind of skipped it for now.

Note that I did manage to get it to work on integrayed graphics insteqd of Nvidia GPU, so if you really need interactive mode for some test, it might be easier to try it on another "simpler" computer.

The run.py is working as expected, so you might want to look a bit further into that. Also I am running run.py on mp3d, not replic.

From: YHWmz @.> Sent: Wednesday, July 20, 2022 8:59:35 PM To: facebookresearch/sound-spaces @.> Cc: Rousslan F.J. Dossa @.>; Mention @.> Subject: Re: [facebookresearch/sound-spaces] DDPPO Agent evaluation and video generation (Issue #80)

I also met the "Fatal Python error: (pygame parachute) Segmentation Fault" error when trying to run

python scripts/interactive_demo.py

and

python ss_baselines/av_nav/run.py --exp-config ss_baselines/av_nav/config/audionav/replica/train_telephone/audiogoal_depth_ddppo.yaml --model-dir data/models/ss2/replica/dav_nav CONTINUOUS True

@dosssmanhttps://github.com/dosssman Do you fix this error now?

— Reply to this email directly, view it on GitHubhttps://github.com/facebookresearch/sound-spaces/issues/80#issuecomment-1190185707, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AFB5WUE4O34WHTEBG4ORTNLVU7S2PANCNFSM537DQSMA. You are receiving this because you were mentioned.Message ID: @.***>

dosssman commented 2 years ago

run.py dors not use pygame, so you should not have that error there.

Are you sure it is the same error than interactive_demo.py ?

From: YHWmz @.> Sent: Wednesday, July 20, 2022 8:59:35 PM To: facebookresearch/sound-spaces @.> Cc: Rousslan F.J. Dossa @.>; Mention @.> Subject: Re: [facebookresearch/sound-spaces] DDPPO Agent evaluation and video generation (Issue #80)

I also met the "Fatal Python error: (pygame parachute) Segmentation Fault" error when trying to run

python scripts/interactive_demo.py

and

python ss_baselines/av_nav/run.py --exp-config ss_baselines/av_nav/config/audionav/replica/train_telephone/audiogoal_depth_ddppo.yaml --model-dir data/models/ss2/replica/dav_nav CONTINUOUS True

@dosssmanhttps://github.com/dosssman Do you fix this error now?

— Reply to this email directly, view it on GitHubhttps://github.com/facebookresearch/sound-spaces/issues/80#issuecomment-1190185707, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AFB5WUE4O34WHTEBG4ORTNLVU7S2PANCNFSM537DQSMA. You are receiving this because you were mentioned.Message ID: @.***>

YHWmz commented 2 years ago

@dosssman I found out that both of them are not pygame errors, but env.reset(). Maybe I will open a new issue to describe it. Thank you for your reply.

run.py dors not use pygame, so you should not have that error there. Are you sure it is the same error than interactive_demo.py ? … ____ From: YHWmz @.> Sent: Wednesday, July 20, 2022 8:59:35 PM To: facebookresearch/sound-spaces @.> Cc: Rousslan F.J. Dossa @.>; Mention @.> Subject: Re: [facebookresearch/sound-spaces] DDPPO Agent evaluation and video generation (Issue #80) I also met the "Fatal Python error: (pygame parachute) Segmentation Fault" error when trying to run python scripts/interactive_demo.py and python ss_baselines/av_nav/run.py --exp-config ss_baselines/av_nav/config/audionav/replica/train_telephone/audiogoal_depth_ddppo.yaml --model-dir data/models/ss2/replica/dav_nav CONTINUOUS True @dosssmanhttps://github.com/dosssman Do you fix this error now? — Reply to this email directly, view it on GitHub<#80 (comment)>, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AFB5WUE4O34WHTEBG4ORTNLVU7S2PANCNFSM537DQSMA. You are receiving this because you were mentioned.Message ID: @.***>

ChanganVR commented 2 years ago

@dosssman did you look at the README page where there are instructions for running evaluation and generating videos for a given checkpoint?

Regarding the interactive demo, the behavior has been quite unpredictable :( It might be because of pygame or hardware compatability issues.

dosssman commented 2 years ago

Hello @ChanganVR Thanks for the answer.

I have indeed tried the commands in the ss_baselines/av_nav/README.md. Since I trained using the command line provided in the repository root README.md:

python ss_baselines/av_nav/run.py --exp-config ss_baselines/av_nav/config/audionav/mp3d/train_telephone/audiogoal_depth_ddppo.yaml --model-dir data/models/ss2/mp3d/dav_nav CONTINUOUS True

I adapted the command for evaluation like the following:

python ss_baselines/av_nav/run.py --run-type eval --exp-config ss_baselines/av_nav/config/audionav/mp3d/train_telephone/audiogoal_depth_ddppo.yaml EVAL_CKPT_PATH_DIR data/models/ss2/mp3d/dav_nav/data/ckpt.0.pth CONTINUOUS True

It raises a NotImplemetedError though:

Traceback (most recent call last):
  File "/home/rousslan/random/rl/ss-hab-headless-py39/sound-spaces/ss_baselines/av_nav/run.py", line 101, in <module>
    main()
  File "/home/rousslan/random/rl/ss-hab-headless-py39/sound-spaces/ss_baselines/av_nav/run.py", line 97, in main
    trainer.eval(args.eval_interval, args.prev_ckpt_ind, config.USE_LAST_CKPT)
  File "/home/rousslan/random/rl/ss-hab-headless-py39/sound-spaces/ss_baselines/common/base_trainer.py", line 105, in eval
    result = self._eval_checkpoint(self.config.EVAL_CKPT_PATH_DIR, writer)
  File "/home/rousslan/random/rl/ss-hab-headless-py39/sound-spaces/ss_baselines/common/base_trainer.py", line 176, in _eval_checkpoint
    raise NotImplementedError

Is DDPPO agent evaluation supported ?

ChanganVR commented 2 years ago

Hi @dosssman please update the config file to --exp-config ss_baselines/av_nav/config/audionav/mp3d/test_telephone/audiogoal_depth.yaml at test time, which uses ppo_trainer.py to evaluate. DDPPO is only for training.

dosssman commented 2 years ago

Hello @ChanganVR . Thanks for pointing out the correct config path.

I adapted the evaluation scripts as follows to get it to run evaluate the trained agent over the mp3d audionav task as follows:

python ss_baselines/av_nav/run.py --run-type eval --exp-config ss_baselines/av_nav/config/audionav/mp3d/test_telephone/audiogoal_depth.yaml EVAL_CKPT_PATH_DIR data/models/ss2/mp3d/dav_nav/data/ckpt.100.pth CONTINUOUS True

and got the following error:

Traceback (most recent call last):
  File "/home/rousslan/random/rl/ss-hab-headless-py39/sound-spaces/ss_baselines/av_nav/run.py", line 101, in <module>
    main()  File "/home/rousslan/random/rl/ss-hab-headless-py39/sound-spaces/ss_baselines/av_nav/run.py", line 97, in main    trainer.eval(args.eval_interval, args.prev_ckpt_ind, config.USE_LAST_CKPT)
  File "/home/rousslan/random/rl/ss-hab-headless-py39/sound-spaces/ss_baselines/common/base_trainer.py", line 105, in eval
    result = self._eval_checkpoint(self.config.EVAL_CKPT_PATH_DIR, writer)
  File "/home/rousslan/random/rl/ss-hab-headless-py39/sound-spaces/ss_baselines/av_nav/ppo/ppo_trainer.py", line 555, in _eval_checkpoint    audios[i].append(observations[i]['audiogoal'])KeyError: 'audiogoal'
  0%|                                                                                             | 0/1000 [00:02<?, ?it/s]
Exception ignored in: <function VectorEnv.__del__ at 0x7ff0b9a89d30>
Traceback (most recent call last):
  File "/home/rousslan/random/rl/ss-hab-headless-py39/habitat-lab/habitat/core/vector_env.py", line 592, in __del__
    self.close()
  File "/home/rousslan/random/rl/ss-hab-headless-py39/habitat-lab/habitat/core/vector_env.py", line 463, in close
    write_fn((CLOSE_COMMAND, None))
  File "/home/rousslan/random/rl/ss-hab-headless-py39/habitat-lab/habitat/core/vector_env.py", line 118, in __call__
    self.write_fn(data)
  File "/home/rousslan/random/rl/ss-hab-headless-py39/habitat-lab/habitat/utils/pickle5_multiprocessing.py", line 62, in send
    self.send_bytes(buf.getvalue())
  File "/home/rousslan/anaconda3/envs/ss-hab-headless-py39/lib/python3.9/multiprocessing/connection.py", line 205, in send_bytes
    self._send_bytes(m[offset:offset + size])
  File "/home/rousslan/anaconda3/envs/ss-hab-headless-py39/lib/python3.9/multiprocessing/connection.py", line 416, in _send_bytes
    self._send(header + buf)
  File "/home/rousslan/anaconda3/envs/ss-hab-headless-py39/lib/python3.9/multiprocessing/connection.py", line 373, in _send
    n = write(self._handle, buf)
BrokenPipeError: [Errno 32] Broken pipe

As a fix, I had to change the sound-spaces/configs/audionav/av_nav/mp3d/audiogoal.yaml configuration file to enable the AUDIOGOAL_SENSOR:

TASK:
  TYPE: AudioNav
  SUCCESS_DISTANCE: 1.0

  # Original
  # SENSORS: ['SPECTROGRAM_SENSOR']
  # GOAL_SENSOR_UUID: spectrogram

  # For eval support
  SENSORS: ['AUDIOGOAL_SENSOR', 'SPECTROGRAM_SENSOR']
  GOAL_SENSOR_UUID: spectrogram # audiogoal

Video with depth sensor based images, as well as the audio were properly generated. With some fiddling, I think I could make the RGB based video plotting work too. Thanks a lot for the instructions to make this work.

PS: Moviepy related error When using torch==1.12.0 installed from conda as per the official website, and moviepy==2.0.0.dev2 installed from PyPI, the TypeError: write_gif() got an unexpected keyword argument 'verbose' might be raised.

A work around was to edit the make_video in /path/to/venv/lib/python3.9/site-packages/torch/utils/tensorboard/summary.py to add the case when moviepy does not support the verbose argument:

def make_video(tensor, fps):
    try:
        import moviepy  # noqa: F401
    except ImportError:
        print("add_video needs package moviepy")
        return
    try:
        from moviepy import editor as mpy
    except ImportError:
        print(
            "moviepy is installed, but can't import moviepy.editor.",
            "Some packages could be missing [imageio, requests]",
        )
        return
    import tempfile

    t, h, w, c = tensor.shape

    # encode sequence of images into gif string
    clip = mpy.ImageSequenceClip(list(tensor), fps=fps)

    filename = tempfile.NamedTemporaryFile(suffix=".gif", delete=False).name
    try:  # newer version of moviepy use logger instead of progress_bar argument.
        clip.write_gif(filename, verbose=False, logger=None)
    except TypeError:
        try:  # older version of moviepy does not support progress_bar argument.
            clip.write_gif(filename, verbose=False, progress_bar=False)
        except TypeError:
            try: # in case verebose argument is also not supported
                clip.write_gif(filename, verbose=False)
            except TypeError:
                clip.write_gif(filename)
zhibinQiu commented 2 years ago

I also met the "Fatal Python error: (pygame parachute) Segmentation Fault" error when trying to run

python scripts/interactive_demo.py

and

python ss_baselines/av_nav/run.py --exp-config ss_baselines/av_nav/config/audionav/replica/train_telephone/audiogoal_depth_ddppo.yaml --model-dir data/models/ss2/replica/dav_nav CONTINUOUS True

@dosssman Have you fixed this error? If so, could you tell me how to solve it?

I got the error too, Could you give me some help to solve it?

dosssman commented 2 years ago

@2206411193 Greetings.

For which script do you get the error ? interactive_demo.py or something else ?

zhibinQiu commented 2 years ago

@dosssman Thanks for your reply. When I try to run 

python ss_baselines/av_nav/run.py --exp-config ss_baselines/av_nav/config/audionav/replica/train_telephone/audiogoal_depth_ddppo.yaml --model-dir data/models/ss2/replica/dav_nav CONTINUOUS True

and got the following error

 File "/ssdhome/qzb521/qinghua/sound-spaces/soundspaces/continuous_simulator.py", line 365 in reset
  File "/ssdhome/qzb521/qinghua/habitat-lab/habitat/core/embodied_task.py", line 284 in reset
  File "/ssdhome/qzb521/qinghua/habitat-lab/habitat/core/env.py", line 252 in reset
  File "/ssdhome/qzb521/qinghua/habitat-lab/habitat/core/env.py", line 402 in reset
  File "/home/qzb521/miniconda3/envs/qinghua/lib/python3.7/contextlib.py", line 74 in inner
  File "/ssdhome/qzb521/qinghua/sound-spaces/ss_baselines/common/environments.py", line 61 in reset
  File "/ssdhome/qzb521/qinghua/habitat-lab/habitat/core/vector_env.py", line 262 in _worker_env
  File "/home/qzb521/miniconda3/envs/qinghua/lib/python3.7/contextlib.py", line 74 in inner
  File "/home/qzb521/miniconda3/envs/qinghua/lib/python3.7/multiprocessing/process.py", line 99 in run
  File "/home/qzb521/miniconda3/envs/qinghua/lib/python3.7/multiprocessing/process.py", line 297 in _bootstrap
  File "/home/qzb521/miniconda3/envs/qinghua/lib/python3.7/multiprocessing/spawn.py", line 118 in _main
  File "/home/qzb521/miniconda3/envs/qinghua/lib/python3.7/multiprocessing/forkserver.py", line 317 in _serve_one
  File "/home/qzb521/miniconda3/envs/qinghua/lib/python3.7/multiprocessing/forkserver.py", line 281 in main
  File "<string>", line 1 in <module>
Fatal Python error: (pygame parachute) Segmentation Fault
dosssman commented 2 years ago

@2206411193

I see.

In my case, I only got the error when trying to run interactive_demon.py. But run.py works as expected in my case.

The main difference I see is that you are using the replica dataset, while I use mp3d aka matterport. I would test with replica on my side to see if I get the same error, but unfortunately I am already running run.py with mp3d dataset and I am afraid to break some things.

Could you try with the mp3d instead ? In that case, I might provide better guidance to get it to work. Another thing that might be helpful would be to know the exact steps you follow to install habitat-lab, habitat-sim, and sound-spaces (which version, etc...).

zhibinQiu commented 2 years ago

@2206411193

I see.

In my case, I only got the error when trying to run interactive_demon.py. But run.py works as expected in my case.

The main difference I see is that you are using the replica dataset, while I use mp3d aka matterport. I would test with replica on my side to see if I get the same error, but unfortunately I am already running run.py with mp3d dataset and I am afraid to break some things.

Could you try with the mp3d instead. In that case, I might provide better guidance to get it to work. Another thing that might be helpful would be to know the exact steps you follow to install habitat-lab, habitat-sim, and sound-spaces (which version, etc...).

Thanks. I will try with the mp3d dataset. and I use the Soundspaces by the the following steps: 1.install habitat-lab 0.2.2 

git clone --branch stable https://github.com/facebookresearch/habitat-lab.git
cd habitat-lab
pip install -e .

2.build the habitat-sim 0.2.2 from source code

git clone --branch stable https://github.com/facebookresearch/habitat-sim.git
cd habitat-sim
conda create -n xxx python=3.7 cmake=3.14.0
conda activate habitat
pip install -r requirements.txt
python setup.py install --with-cuda --bullet --headless --audio

3.install Soundspace

git clone https://github.com/facebookresearch/sound-spaces.git
pip install -e .

and my system environment: Ubuntu 20.04.3 LTS python 3.7.13

dosssman commented 2 years ago

@2206411193

Roger that.

Installation process looks similar to mine. The author @ChanganVR has recently update the instructions for building habitat-sim for soundspace. Instead of git clone --branch stable https://github.com/facebookresearch/habitat-sim.git, I would recommend using habitat-sim from this commit.

One more thing, for mp3d, I had to do some pretty intricate steps to get the essentially data downloaded, as well as skip the heavy datasets file that are required by soundspaces 1.0, but not soundspaces 2.0.

I have compiled all the steps that got it working on my side as follows. Hope it helps:

Soundspaces - Habitat-lab - Habitat-sim setup

General guideline as of 2022-07-21

System specifics

conda create -n ss-hab-headless-py39 python=3.9 cmake=3.14.0 -y
conda activate ss-hab
pip install pytest-xdist

Habitat-lab Stable 0.2.2

git clone --branch stable https://github.com/facebookresearch/habitat-lab.git # Currently @ 0f454f62e41050bc90ca468c62db35d7484923ff
cd habitat-lab
# pip install -e .
# While installing deps., pip threw out error that some conflict due to TF 2.1.0 was preventing installing all the deps.
# Manually ran `pip install tensorflow-gpu==2.1.0` just to be sure
# Additionally, install all the other deps for dev. purposes
pip install -r requirements.txt
python setup.py develop --all # So far so good

# Leave the directory for the subsequent steps
cd ..

Habitat-sim

# Makes sure all system level deps are installed.
sudo apt-get update || True
sudo apt-get install -y --no-install-recommends libjpeg-dev libglm-dev libgl1-mesa-glx libegl1-mesa-dev mesa-utils xorg-dev freeglut3-dev # Ubuntu
# Clone habitat-sim repository
git clone https://github.com/facebookresearch/habitat-sim.git # Current @ 80f8e31140eaf50fe6c5ab488525ae1bdf250bd9
cd habitat-sim
# Checkout the commit suggest by ChanganVR
git checkout 80f8e31140eaf50fe6c5ab488525ae1bdf250bd9
# Build habitat-sim with audio and headless support
python setup.py install --with-cuda --bullet --audio --headless # compilation goes brrrr...
pip install hypothesis # For the tests mainly

Additional building instructions

Preliminary testing

__Getting the test scene_datasets__:

Some tests (tests/test_physics.py, tests/test_simlulator.py) require "habitat test scenes" and "habitat test objects". Where are those ? Datasets ? The provided download tool is broken. Manually downloading those:

To get the habitat-test-scenes, from the habitat-sim root folder:

python src_python/habitat_sim/utils/datasets_download.py --uids habitat_test_scenes

Getting the (test / example) habitat objects:

python src_python/habitat_sim/utils/datasets_download.py --uids habitat_example_objects

With this, pytest tests/test_physics.py should have 100% success rate.

Interactive testing

This assumes habitat-sim was built with display support.

python examples/viewer.py --scene data/scene_datasets/habitat-test-scenes/skokloster-castle.glb

Note: In case it return ModuleNotFound examples.settings error, edit the examples/viewer.py and remove the examples. from the relevant import line.

Non-interactive testing

With compiled habitat-sim, this fails to run, returning a free(): invalid pointer. Should work if run from habitat-lab directory instead of habitat-sim.

python examples/example.py --scene data/scene_datasets/habitat-test-scenes/skokloster-castle.glb

Acquiring datasets necessary for simulations

Getting the ReplicaCAD dataset

python src_python/habitat_sim/utils/datasets_download.py --uids replica_cad_dataset

Running the physics interaction simulation:

python examples/viewer.py --dataset data/replica_cad/replicaCAD.scene_dataset_config.json --scene apt_1

Other notes

To check that both habitat-lab and habitat-sim work with each other:

cd ../habitat-lab
python examples/example.py

It should say "... ran for 200 steps" or something similar.

Soundspaces 2.0

# git clone https://github.com/facebookresearch/sound-spaces.git # Currently @ fb68e410a4a1388e2d63279e6b92b6f082371fec
git clone https://github.com/facebookresearch/sound-spaces.git
cd sound-spaces
git checkout fb68e410a4a1388e2d63279e6b92b6f082371fec
pip install -e .

Downloading the scene_datasets for SS 2.0 habitat audio visual simulations

Requires access to the download_mp.py tool from official Matterport3D. See https://github.com/facebookresearch/habitat-lab#matterport3d

mkdir -p data/scene_datasets
mkdir -p data/versioned_data/mp3d
python /path/to/download_mp.py --task habitat -o /path/to/sound-spaces/data/versioned_data/mp3d

This will download a ZIP file into /path/to/sound-spaces/data/versioned_data/mp3d/v1/tasks/mp3d_habitat.zip

Unzip this file to obtain /path/to/sound-spaces/data/versioned_data/mp3d/v1/tasks/mp3d. This folder should contain files like: 17DRP5sb8fy, 1LXtFkjw3qL, etc...

Make it so that /path/to/sound-spaces/data/scene_datasets/mp3d points to /path/to/sound-spaces/data/versioned_data/mp3d/v1/tasks/mp3d. For example:

ln -s /path/to/sound-spaces/data/versioned_data/mp3d/v1/tasks/mp3d` `/path/to/sound-spaces/data/scene_datasets/mp3d`

Some additional metadata that is intertwined with other datasets and features of soundspaces is also required:

# From /path/to/soundspaces/data, run:
wget http://dl.fbaipublicfiles.com/SoundSpaces/metadata.tar.xz && tar xvf metadata.tar.xz # 1M
wget http://dl.fbaipublicfiles.com/SoundSpaces/sounds.tar.xz && tar xvf sounds.tar.xz #13M
wget http://dl.fbaipublicfiles.com/SoundSpaces/datasets.tar.xz && tar xvf datasets.tar.xz #77M
wget http://dl.fbaipublicfiles.com/SoundSpaces/pretrained_weights.tar.xz && tar xvf 
pretrained_weights.tar.xz
# This heavy file can be ignored for SS 2.0
# wget http://dl.fbaipublicfiles.com/SoundSpaces/binaural_rirs.tar && tar xvf binaural_rirs.tar # 867G

SS 2.0 command provided in the Reamde are based on mp3d datasets. If trying to run the interactive mode command latter on, it will likely throw up some error about data/metadata/default/... being absent.

This will require the following tweak:

Downloading mp3d_material_config.json

Download from the following link: https://github.com/facebookresearch/rlr-audio-propagation/blob/main/RLRAudioPropagationPkg/data/mp3d_material_config.json and put it in /path/to/soundspaces/data/.

Torch

The torch install that comes with the dependencies should work by default on something like GTX 1080 Ti. However, because that one relies on CUDA 10.2 it cannot be used with an RTX 3090 for example (CUDA Error: no kernel image is available for execution on the device ...). Training on an RTX 3090 as of 2022-07-21, thus requires upgrading to a torch version that supports CUDA 11.

conda install pytorch torchvision torchaudio cudatoolkit=11.3 -c pytorch

This will install torch==1.12.0.

CUDA 11.6 unfortunately seems to create too many conflicts with other dependencies, solving the environment ad infinitum.

conda install pytorch torchvision torchaudio cudatoolkit=11.3 -c pytorch

Testing SS2.0 in interactive mode

For a machine with display, and with habitat-sim not being built with the --headless flag.

python scripts/interactive_mode.py

Note: This worked on the internal graphics of the motherboard, but not on the RTX 3090 GPU. It might work on an older, or not so fancy GPU. In any case, interactive mode is not that important for the RL use case.

[New] Training continuous navigation agent DDPPO baseline

python ss_baselines/av_nav/run.py --exp-config ss_baselines/av_nav/config/audionav/mp3d/train_telephone/audiogoal_depth_ddppo.yaml --model-dir data/models/ss2/mp3d/dav_nav CONTINUOUS True

Training continuous navigation PPO baseline

python ss_baselines/av_nav/run.py --exp-config ss_baselines/av_nav/config/audionav/mp3d/train_telephone/audiogoal_depth_ppo.yaml --model-dir data/models/ss2/mp3d/dav_nav_ppo/ CONTINUOUS True

Evulating the trained agent

This is done using the test confgurations suggested in the sound-spaces/ss_baselines/av_nav/README.md file.

python ss_baselines/av_nav/run.py --run-type eval --exp-config ss_baselines/av_nav/config/audionav/mp3d/test_telephone/audiogoal_depth.yaml EVAL_CKPT_PATH_DIR data/models/ss2/mp3d/dav_nav/data/ckpt.100.pth CONTINUOUS True

Missing audiogoal in observations error

Runnin the commnad above will probably spit out an error related to the audiogal field missing in the observations directly. The fix is to change the base task configuration with the following:

TASK:
  TYPE: AudioNav
  SUCCESS_DISTANCE: 1.0

  # Original
  # SENSORS: ['SPECTROGRAM_SENSOR']
  # GOAL_SENSOR_UUID: spectrogram

  # For eval support
  SENSORS: ['AUDIOGOAL_SENSOR', 'SPECTROGRAM_SENSOR']
  GOAL_SENSOR_UUID: spectrogram # audiogoal

Basically, it adds the AUDIOGOAL_SENSOR to the config, which in turns generates the corresponding field in the observation of the agent

TypeError: write_gif() got an unexpected keyword argument 'verbose'

Best guess is some form of mismatch between the moviepy version that the tensorboard used here expectes and the one that is actually installed. Current versions are torch==1.12.0 installed from conda according to the official pytorch website, and moviepy==2.0.0.dev2 install from PyPI.

A work around was to edit the make_video in /path/to/venv/lib/python3.9/site-packages/torch/utils/tensorboard/summary.py to add the case when moviepy does not support the verbose argument:

def make_video(tensor, fps):
    try:
        import moviepy  # noqa: F401
    except ImportError:
        print("add_video needs package moviepy")
        return
    try:
        from moviepy import editor as mpy
    except ImportError:
        print(
            "moviepy is installed, but can't import moviepy.editor.",
            "Some packages could be missing [imageio, requests]",
        )
        return
    import tempfile

    t, h, w, c = tensor.shape

    # encode sequence of images into gif string
    clip = mpy.ImageSequenceClip(list(tensor), fps=fps)

    filename = tempfile.NamedTemporaryFile(suffix=".gif", delete=False).name
    try:  # newer version of moviepy use logger instead of progress_bar argument.
        clip.write_gif(filename, verbose=False, logger=None)
    except TypeError:
        try:  # older version of moviepy does not support progress_bar argument.
            clip.write_gif(filename, verbose=False, progress_bar=False)
        except TypeError:
            try: # in case verebose argument is also not supported
                clip.write_gif(filename, verbose=False)
            except TypeError:
                clip.write_gif(filename)
zhibinQiu commented 2 years ago

@dosssman Thanks a million. 👍

dosssman commented 2 years ago

Using the Matterplot3D dataset requires some permissiosn to access the data. Unless you have already obtained it, you should probably check here: https://github.com/facebookresearch/habitat-lab#matterport3d to apply for the permission first, then mess around the code while waiting for their answer.

zhibinQiu commented 2 years ago

Using the Matterplot3D dataset requires some permissiosn to access the data. Unless you have already obtained it, you should probably check here: https://github.com/facebookresearch/habitat-lab#matterport3d to apply for the permission first, then mess around the code while waiting for their answer.

They have rejected me because my account is free, I am still a student and cannot afford to pay for a subscription.😭

dosssman commented 2 years ago

No way, I am also a student and I did not have to pay for a subscription. The dataset is free as long as it is in the scope of doing research, and not commercial activities.

How about you try the procedure underlined under the Data section here: https://github.com/niessner/Matterport You have to fill a PDF form and send it to the e-mail address specified, and it will give you access to the download_mp.py script that I also used to get soundspaces running over here. Make sure to specify your university and research affiliation it should be all right.

zhibinQiu commented 2 years ago

@dosssman

Hi, according to your instructions I successfully run soundspaces2.0 on Replica. There are only two small problems will occur in continuous_simulator.py that need to be fixed, I don't know if other people have got these problems.

  1. Error occur in channel_layout.channelType = habitat_sim.sensor.RLRAudioPropagationChannelLayoutType.Binaural The error tell me habitat_sim.sensor don't have the attribution 'RLRAudioPropagationChannelLayoutType' I add it in Hatbitat-Sim's source and rebuild.

  2. Aorting when load semantics. set ' acoustics_config.enableMaterials = False' in continuous_simulator.py I'm new to Soundspace, so I don't quite understand what it means.

Thanks again for your help.

dosssman commented 2 years ago

@2206411193

Hello there. Glad it could be of some help!

Can you also run soundspace's interactive_mode.py successfully after this fix ?

zhibinQiu commented 2 years ago

@dosssman My Linux has no GUI, can I run this program?

dosssman commented 2 years ago

@2206411193

Nevermind then haha. I will try to make some time and try on my side.

Thanks for the update. Best wishes for your experiments.