Broken Pipe arising from data missing

[rgoggins]

Hi,

I wanted to raise an issue because I am having a great deal of difficulty finding the correct gibson dataset.

I am getting a broken pipe while running python3 main.py -n1 --auto_gpu_config 0 --split val due to being unable to load the scene_dataset Cantwell.glb.

Here is the stacktrace for reference:

(venv) ryangoggins@Ryans-MBP-2 Neural-SLAM % python3 main.py -n1 --auto_gpu_config 0 --split val
Dumping at ./tmp//models/exp1/
Namespace(alpha=0.99, auto_gpu_config=0, camera_height=1.25, clip_param=0.2, collision_threshold=0.2, cuda=False, du_scale=2, dump_location='./tmp/', entropy_coef=0.001, env_frame_height=256, env_frame_width=256, eps=1e-05, eval=0, exp_loss_coeff=1.0, exp_name='exp1', frame_height=128, frame_width=128, gamma=0.99, global_downscaling=2, global_hidden_size=256, global_lr=2.5e-05, goals_size=2, hfov=90.0, load_global='0', load_local='0', load_slam='0', local_hidden_size=512, local_optimizer='adam,lr=0.0001', local_policy_update_freq=5, log_interval=10, map_pred_threshold=0.5, map_resolution=5, map_size_cm=2400, max_episode_length=1000, max_grad_norm=0.5, no_cuda=False, noise_level=1.0, noisy_actions=1, noisy_odometry=1, num_episodes=1000000, num_global_steps=40, num_local_steps=25, num_mini_batch=0, num_processes=1, num_processes_on_first_gpu=0, num_processes_per_gpu=11, obs_threshold=1, obstacle_boundary=5, pose_loss_coeff=10000.0, ppo_epoch=4, pretrained_resnet=1, print_images=0, proj_loss_coeff=1.0, randomize_env_every=1000, save_interval=1, save_periodic=500000, save_trajectory_data='0', seed=1, short_goal_dist=1, sim_gpu_id=0, slam_batch_size=72, slam_iterations=10, slam_memory_size=500000, slam_optimizer='adam,lr=0.0001', split='val', task_config='tasks/pointnav_gibson.yaml', tau=0.95, total_num_scenes='auto', train_global=1, train_local=1, train_slam=1, use_deterministic_local=0, use_gae=False, use_pose_estimation=2, use_recurrent_global=0, use_recurrent_local=1, value_loss_coef=0.5, vis_type=1, vision_range=64, visualize=0)
Loading data/scene_datasets/gibson/Cantwell.glb
2022-07-19 19:16:44,274 initializing sim Sim-v0
Renderer: AMD Radeon Pro 5500M OpenGL Engine by ATI Technologies Inc.
OpenGL version: 4.1 ATI-4.8.54
Using optional features:
    GL_ARB_ES2_compatibility
    GL_ARB_separate_shader_objects
    GL_ARB_texture_filter_anisotropic
    GL_ARB_texture_storage
    GL_ARB_vertex_array_object
    GL_EXT_debug_label
    GL_EXT_debug_marker
Using driver workarounds:
    no-layout-qualifiers-on-old-glsl
WARNING: Logging before InitGoogleLogging() is written to STDERR
E0719 19:16:44.408632 90494464 ResourceManager.cpp:62] Cannot load from file data/scene_datasets/gibson/Cantwell.glb
E0719 19:16:44.408958 90494464 Simulator.cpp:106] cannot load data/scene_datasets/gibson/Cantwell.glb
I0719 19:16:44.409894 90494464 SceneManager.h:24] Deconstructing SceneManager
I0719 19:16:44.409978 90494464 SceneGraph.h:20] Deconstructing SceneGraph
I0719 19:16:44.410568 90494464 Renderer.cpp:38] Deconstructing Renderer
I0719 19:16:44.410748 90494464 WindowlessContext.cpp:257] Deconstructing GL context
Process ForkServerProcess-1:
Traceback (most recent call last):
  File "/usr/local/Cellar/python@3.7/3.7.13_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
  File "/usr/local/Cellar/python@3.7/3.7.13_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/multiprocessing/process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "/Users/ryangoggins/Desktop/arbitrary_code/drone/Neural-SLAM/env/habitat/habitat_api/habitat/core/vector_env.py", line 148, in _worker_env
    env = env_fn(*env_fn_args)
  File "/Users/ryangoggins/Desktop/arbitrary_code/drone/Neural-SLAM/env/habitat/__init__.py", line 22, in make_env_fn
    config_env=config_env, config_baseline=config_baseline, dataset=dataset
  File "/Users/ryangoggins/Desktop/arbitrary_code/drone/Neural-SLAM/env/habitat/exploration_env.py", line 83, in __init__
    super().__init__(config_env, dataset)
  File "/Users/ryangoggins/Desktop/arbitrary_code/drone/venv/src/habitat/habitat/core/env.py", line 290, in __init__
    self._env = Env(config, dataset)
  File "/Users/ryangoggins/Desktop/arbitrary_code/drone/venv/src/habitat/habitat/core/env.py", line 93, in __init__
    id_sim=self._config.SIMULATOR.TYPE, config=self._config.SIMULATOR
  File "/Users/ryangoggins/Desktop/arbitrary_code/drone/venv/src/habitat/habitat/sims/registration.py", line 19, in make_sim
    return _sim(**kwargs)
  File "/Users/ryangoggins/Desktop/arbitrary_code/drone/venv/src/habitat/habitat/sims/habitat_simulator/habitat_simulator.py", line 160, in __init__
    self._sim = habitat_sim.Simulator(self.sim_config)
  File "<attrs generated init habitat_sim.simulator.Simulator>", line 7, in __init__
    self.__attrs_post_init__()
  File "/Users/ryangoggins/Desktop/arbitrary_code/drone/venv/lib/python3.7/site-packages/habitat_sim-0.1.2-py3.7-macosx-12-x86_64.egg/habitat_sim/simulator.py", line 38, in __attrs_post_init__
    self.reconfigure(config)
  File "/Users/ryangoggins/Desktop/arbitrary_code/drone/venv/lib/python3.7/site-packages/habitat_sim-0.1.2-py3.7-macosx-12-x86_64.egg/habitat_sim/simulator.py", line 104, in reconfigure
    self._config_backend(config)
  File "/Users/ryangoggins/Desktop/arbitrary_code/drone/venv/lib/python3.7/site-packages/habitat_sim-0.1.2-py3.7-macosx-12-x86_64.egg/habitat_sim/simulator.py", line 56, in _config_backend
    self._sim = hsim.SimulatorBackend(config.sim_cfg)
ValueError: Cannot load: data/scene_datasets/gibson/Cantwell.glb
Traceback (most recent call last):
  File "main.py", line 769, in <module>
    main()
  File "main.py", line 119, in main
    envs = make_vec_envs(args)
  File "/Users/ryangoggins/Desktop/arbitrary_code/drone/Neural-SLAM/env/__init__.py", line 7, in make_vec_envs
    envs = construct_envs(args)
  File "/Users/ryangoggins/Desktop/arbitrary_code/drone/Neural-SLAM/env/habitat/__init__.py", line 102, in construct_envs
    range(args.num_processes))
  File "/Users/ryangoggins/Desktop/arbitrary_code/drone/Neural-SLAM/env/habitat/habitat_api/habitat/core/vector_env.py", line 117, in __init__
    read_fn() for read_fn in self._connection_read_fns
  File "/Users/ryangoggins/Desktop/arbitrary_code/drone/Neural-SLAM/env/habitat/habitat_api/habitat/core/vector_env.py", line 117, in <listcomp>
    read_fn() for read_fn in self._connection_read_fns
  File "/usr/local/Cellar/python@3.7/3.7.13_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/multiprocessing/connection.py", line 250, in recv
    buf = self._recv_bytes()
  File "/usr/local/Cellar/python@3.7/3.7.13_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/multiprocessing/connection.py", line 407, in _recv_bytes
    buf = self._recv(4)
  File "/usr/local/Cellar/python@3.7/3.7.13_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/multiprocessing/connection.py", line 383, in _recv
    raise EOFError
EOFError
Exception ignored in: <function VectorEnv.__del__ at 0x1428053b0>
Traceback (most recent call last):
  File "/Users/ryangoggins/Desktop/arbitrary_code/drone/Neural-SLAM/env/habitat/habitat_api/habitat/core/vector_env.py", line 487, in __del__
    self.close()
  File "/Users/ryangoggins/Desktop/arbitrary_code/drone/Neural-SLAM/env/habitat/habitat_api/habitat/core/vector_env.py", line 351, in close
    write_fn((CLOSE_COMMAND, None))
  File "/usr/local/Cellar/python@3.7/3.7.13_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/multiprocessing/connection.py", line 206, in send
    self._send_bytes(_ForkingPickler.dumps(obj))
  File "/usr/local/Cellar/python@3.7/3.7.13_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/multiprocessing/connection.py", line 404, in _send_bytes
    self._send(header + buf)
  File "/usr/local/Cellar/python@3.7/3.7.13_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/multiprocessing/connection.py", line 368, in _send
    n = write(self._handle, buf)
BrokenPipeError: [Errno 32] Broken pipe

In order to load the data into Neural-SLAM/, I tried downloading the gibson_tiny, but there was no folder for Cantwell, I then tried downloading the gibson_full dataset and found a Cantwell folder, but the only data the folder contains is camera_poses.csv, mesh_z_up.obj, mesh.obj, a folder titled "pano" and three subfolders "mist", "points" and "rgb". None of these folders contain a .glb or a .navmesh file. I am confused as to what exact zipped data file is the correct one as the Google Form that is listed on the StanfordVL/GibsonEnv repository contains download links for >10 different files and I don't want to download all of these locally.

The current link for downloading Gibson dataset is here: https://github.com/StanfordVL/GibsonEnv/blob/master/gibson/data/README.md

It has links to a Google form that allows you to get access to download links, but none of those links actually appear to have glb files or navmesh files.

Is there a specific download link that I can follow to find the dataset that is required to run the basic main script to test the pretrained models?

[MaLin-art]

I have the same problem, I don't know exactly which dataset file, did you solve your problem?

[rgoggins]

Unfortunately, no. I ended up trying most of them and couldn't find any with glb or navmesh files. If you are able to get any of the datasets working, let me know which link you used.

[MaLin-art]

I am solving this problem these days, if it is solved I will inform you, learn from each other!

------------------ 原始邮件 ------------------ 发件人: "devendrachaplot/Neural-SLAM" @.>; 发送时间: 2022年8月8日(星期一) 中午12:17 @.>; @.**@.>; 主题: Re: [devendrachaplot/Neural-SLAM] Broken Pipe arising from data missing (Issue #59)

Unfortunately, no. I ended up trying most of them and couldn't find any with glb or navmesh files. If you are able to get any of the datasets working, let me know which link you used.

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: @.***>

[rgoggins]

The answer could be that you use the input data from the datasets listed and the habitat-sim repo to generate the files. Take a look here:

https://github.com/facebookresearch/habitat-sim/issues/662

[devendrachaplot]

I am not sure if this issue was resolved, but you need to fill this google form: https://docs.google.com/forms/d/e/1FAIpQLScWlx5Z1DM1M-wTSXaa6zV8lTFkPmTHW1LqMsoCBDWsTDjBkQ/viewform

And then download gibson_habitat_trainval.zip.

[GuoPingPan]

I am solving this problem these days, if it is solved I will inform you, learn from each other! … ------------------ 原始邮件 ------------------ 发件人: "devendrachaplot/Neural-SLAM" @.>; 发送时间: 2022年8月8日(星期一) 中午12:17 @.>; @.**@.>; 主题: Re: [devendrachaplot/Neural-SLAM] Broken Pipe arising from data missing (Issue #59) Unfortunately, no. I ended up trying most of them and couldn't find any with glb or navmesh files. If you are able to get any of the datasets working, let me know which link you used. — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: @.***>

Had you fine the dataset? Can you share with me?