search

facebookresearch / sound-spaces

A first-of-its-kind acoustic simulation platform for audio-visual embodied AI research. It supports training and evaluating multiple tasks and applications.
https://soundspaces.org
Creative Commons Attribution 4.0 International
345 stars 55 forks source link

FileNotFoundError during SAVI training #89

Closed sivannavis closed 2 years ago

sivannavis commented 2 years ago

Hi @ChanganVR! When I run ss_baselines.savi.run with savi_pretraining.yaml, I run into these errors: FileNotFoundError: data/metadata/default/ZMojNkEp431/points.txt not found. Why would the metadata_dir function in simulator.py look for config.SCENE_DATASET = default while in config the value is "mp3d"? I'm running savi/run.py on 1 node with 2 GPUs. I attached the error logs below. Thank you so much!

INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-22 17:19:35,903 Initializing dataset SemanticAudioNav 2022-08-22 17:19:35,903 Initializing dataset SemanticAudioNav INFO:root:Sampled 0 episodes from 0 scenes. INFO:root:Sampled 0 episodes from 0 scenes. INFO:root:Using VectorEnv INFO:root:Using VectorEnv 2022-08-22 17:19:47,443 Initializing dataset SemanticAudioNav 2022-08-22 17:19:47,443 Initializing dataset SemanticAudioNav 2022-08-22 17:19:47,472 Initializing dataset SemanticAudioNav 2022-08-22 17:19:47,473 Initializing dataset SemanticAudioNav 2022-08-22 17:19:47,474 Initializing dataset SemanticAudioNav 2022-08-22 17:19:47,475 Initializing dataset SemanticAudioNav 2022-08-22 17:19:47,530 Initializing dataset SemanticAudioNav 2022-08-22 17:19:47,752 Initializing dataset SemanticAudioNav 2022-08-22 17:19:47,828 Initializing dataset SemanticAudioNav 2022-08-22 17:19:48,055 Initializing dataset SemanticAudioNav 2022-08-22 17:19:48,224 Initializing dataset SemanticAudioNav 2022-08-22 17:19:48,283 Initializing dataset SemanticAudioNav 2022-08-22 17:19:48,283 Initializing dataset SemanticAudioNav 2022-08-22 17:19:48,283 Initializing dataset SemanticAudioNav 2022-08-22 17:19:48,283 Initializing dataset SemanticAudioNav 2022-08-22 17:19:48,291 Initializing dataset SemanticAudioNav 2022-08-22 17:19:48, INFO: Sampled 12908 episodes from 7 scenes. 2022-08-22 17:19:48,363 initializing sim SoundSpacesSim Process ForkServerProcess-5: Traceback (most recent call last): File "/ext3/miniconda3/envs/soundspaces/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap self.run() File "/ext3/miniconda3/envs/soundspaces/lib/python3.8/multiprocessing/process.py", line 108, in run self._target(*self._args, self._kwargs) File "/ext3/miniconda3/envs/soundspaces/lib/python3.8/contextlib.py", line 75, in inner return func(*args, *kwds) File "/ext3/code/habitat-lab/habitat/core/vector_env.py", line 233, in _worker_env env = env_fn(env_fn_args) File "/ext3/code/sound-spaces/ss_baselines/common/env_utils.py", line 134, in make_env_fn env = env_class(config=config, dataset=dataset) File "/ext3/code/sound-spaces/ss_baselines/common/environments.py", line 48, in init super().init(self._core_env_config, dataset) File "/ext3/code/habitat-lab/habitat/core/env.py", line 374, in init self._env = Env(config, dataset) File "/ext3/code/habitat-lab/habitat/core/env.py", line 104, in init self._sim = make_sim( File "/ext3/code/habitat-lab/habitat/sims/registration.py", line 19, in make_sim return _sim(kwargs) File "/ext3/code/sound-spaces/soundspaces/simulator.py", line 135, in init self.points, self.graph = load_metadata(self.metadata_dir) File "/ext3/code/sound-spaces/soundspaces/utils.py", line 27, in load_metadata points_data = np.loadtxt(points_file, delimiter="\t") File "/ext3/miniconda3/envs/soundspaces/lib/python3.8/site-packages/numpy/lib/npyio.py", line 1042, in loadtxt fh = np.lib._datasource.open(fname, 'rt', encoding=encoding) File "/ext3/miniconda3/envs/soundspaces/lib/python3.8/site-packages/numpy/lib/_datasource.py", line 193, in open return ds.open(path, mode, encoding=encoding, newline=newline) File "/ext3/miniconda3/envs/soundspaces/lib/python3.8/site-packages/numpy/lib/_datasource.py", line 532, in open raise FileNotFoundError(f"{path} not found.") FileNotFoundError: data/metadata/default/jh4fc5c5qoQ/points.txt not found. 2022-08-22 17:19:49, INFO: Sampled 12908 episodes from 7 scenes. 2022-08-22 17:19:49,209 initializing sim SoundSpacesSim Process ForkServerProcess-5: Traceback (most recent call last): File "/ext3/miniconda3/envs/soundspaces/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap self.run() File "/ext3/miniconda3/envs/soundspaces/lib/python3.8/multiprocessing/process.py", line 108, in run self._target(*self._args, self._kwargs) File "/ext3/miniconda3/envs/soundspaces/lib/python3.8/contextlib.py", line 75, in inner return func(*args, *kwds) File "/ext3/code/habitat-lab/habitat/core/vector_env.py", line 233, in _worker_env env = env_fn(env_fn_args) File "/ext3/code/sound-spaces/ss_baselines/common/env_utils.py", line 134, in make_env_fn env = env_class(config=config, dataset=dataset) File "/ext3/code/sound-spaces/ss_baselines/common/environments.py", line 48, in init super().init(self._core_env_config, dataset) File "/ext3/code/habitat-lab/habitat/core/env.py", line 374, in init self._env = Env(config, dataset) File "/ext3/code/habitat-lab/habitat/core/env.py", line 104, in init self._sim = make_sim( File "/ext3/code/habitat-lab/habitat/sims/registration.py", line 19, in make_sim return _sim(kwargs) File "/ext3/code/sound-spaces/soundspaces/simulator.py", line 135, in init self.points, self.graph = load_metadata(self.metadata_dir) File "/ext3/code/sound-spaces/soundspaces/utils.py", line 27, in load_metadata points_data = np.loadtxt(points_file, delimiter="\t") File "/ext3/miniconda3/envs/soundspaces/lib/python3.8/site-packages/numpy/lib/npyio.py", line 1042, in loadtxt fh = np.lib._datasource.open(fname, 'rt', encoding=encoding) File "/ext3/miniconda3/envs/soundspaces/lib/python3.8/site-packages/numpy/lib/_datasource.py", line 193, in open return ds.open(path, mode, encoding=encoding, newline=newline) File "/ext3/miniconda3/envs/soundspaces/lib/python3.8/site-packages/numpy/lib/_datasource.py", line 532, in open raise FileNotFoundError(f"{path} not found.") FileNotFoundError: data/metadata/default/ZMojNkEp431/points.txt not found.

sivannavis commented 2 years ago

This is still the case when I evaluate the pretrained model using py ss_baselines/savi/run.py --run-type eval --exp-config ss_baselines/savi/config/semantic_audionav/savi.yaml EVAL_CKPT_PATH_DIR data/pretrained_weights/semantic_audionav/savi/best_val.pth EVAL.SPLIT test USE_SYNC_VECENV True RL.DDPPO.pretrained False

After some debugging I noticed config.TASK_CONFIG is good before the make_env_fn function. And it seems it's this part in initializing the Env Class that changes the SCENE_DATASET to "default", which I'm a bit confused about: https://github.com/facebookresearch/habitat-lab/blob/380ac0a7d8c4ead1532f109b15d329473212eae9/habitat/core/env.py#L95

self._config.SIMULATOR.SCENE_DATASET = ( self.current_episode.scene_dataset_config )

https://github.com/facebookresearch/habitat-lab/blob/380ac0a7d8c4ead1532f109b15d329473212eae9/habitat/core/dataset.py#L60

scene_dataset_config: str = attr.ib( default="default", validator=not_none_validator )

Could you help me take a look and point me on the right way?

sivannavis commented 2 years ago

My temporary workaround for this issue is to comment this part: https://github.com/facebookresearch/habitat-lab/blob/380ac0a7d8c4ead1532f109b15d329473212eae9/habitat/core/env.py#L94-L96

Although I don't know the correct way of solving this, so please point out if I did something wrong. Thanks!