search

devendrachaplot / Neural-SLAM

Pytorch code for ICLR-20 Paper "Learning to Explore using Active Neural SLAM"
http://www.cs.cmu.edu/~dchaplot/projects/neural-slam.html
MIT License
750 stars 138 forks source link

RuntimeError: cuDNN error: CUDNN_STATUS_EXECUTION_FAILED + Broken pipe #66

Closed mylad13 closed 1 year ago

mylad13 commented 1 year ago

I have installed the correct versions of habitat-api and habitat-sim following the instructions, and my cuda version is 10.0. I had the same error with the latest version of cuda as well, so I downgraded to 10.0, but I'm still having problems. I can run the benchmark.py and example.py files from the examples directory of habitat-sim with no problems. This is the final result of running benchmark.py:

====== FPS (512 x 512, depth_only): 4176.6 ====== ================ Performance (FPS) NPROC=1 =================================== Resolution rgb rgbd depth_only 128 x 128 4931.0 2417.7 4101.3
256 x 256 4244.3 1632.6 2437.3
512 x 512 2557.2 681.1 906.4

================ Performance (FPS) NPROC=3 =================================== Resolution rgb rgbd depth_only 128 x 128 4764.4 2371.2 4774.2
256 x 256 4422.2 2199.2 4412.6
512 x 512 4904.7 1817.6 2546.0

================ Performance (FPS) NPROC=5 =================================== Resolution rgb rgbd depth_only 128 x 128 4654.2 2347.9 4709.4
256 x 256 4392.7 2199.7 4392.8
512 x 512 4911.2 2415.8 4176.6

However, when I run the command: python main.py -n1 --auto_gpu_config 0 --split val in the Neural-SLAM directory, I get this error after a while:

(habitat) farjadnm@PC-SIROUSPOUR14:~/Neural-SLAM$ python main.py -n1 --auto_gpu_config 0 --split val Dumping at ./tmp//models/exp1/ Namespace(alpha=0.99, auto_gpu_config=0, camera_height=1.25, clip_param=0.2, collision_threshold=0.2, cuda=True, du_scale=2, dump_location='./tmp/', entropy_coef=0.001, env_frame_height=256, env_frame_width=256, eps=1e-05, eval=0, exp_loss_coeff=1.0, exp_name='exp1', frame_height=128, frame_width=128, gamma=0.99, global_downscaling=2, global_hidden_size=256, global_lr=2.5e-05, goals_size=2, hfov=90.0, load_global='0', load_local='0', load_slam='0', local_hidden_size=512, local_optimizer='adam,lr=0.0001', local_policy_update_freq=5, log_interval=10, map_pred_threshold=0.5, map_resolution=5, map_size_cm=2400, max_episode_length=1000, max_grad_norm=0.5, no_cuda=False, noise_level=1.0, noisy_actions=1, noisy_odometry=1, num_episodes=1000000, num_global_steps=40, num_local_steps=25, num_mini_batch=0, num_processes=1, num_processes_on_first_gpu=0, num_processes_per_gpu=11, obs_threshold=1, obstacle_boundary=5, pose_loss_coeff=10000.0, ppo_epoch=4, pretrained_resnet=1, print_images=0, proj_loss_coeff=1.0, randomize_env_every=1000, save_interval=1, save_periodic=500000, save_trajectory_data='0', seed=1, short_goal_dist=1, sim_gpu_id=0, slam_batch_size=72, slam_iterations=10, slam_memory_size=500000, slam_optimizer='adam,lr=0.0001', split='val', task_config='tasks/pointnav_gibson.yaml', tau=0.95, total_num_scenes='auto', train_global=1, train_local=1, train_slam=1, use_deterministic_local=0, use_gae=False, use_pose_estimation=2, use_recurrent_global=0, use_recurrent_local=1, value_loss_coef=0.5, vis_type=1, vision_range=64, visualize=0) Loading data/scene_datasets/gibson/Cantwell.glb 2023-03-07 11:55:31,646 initializing sim Sim-v0 Renderer: NVIDIA GeForce RTX 3090/PCIe/SSE2 by NVIDIA Corporation OpenGL version: 4.6.0 NVIDIA 525.85.05 Using optional features: GL_ARB_ES2_compatibility GL_ARB_direct_state_access GL_ARB_get_texture_sub_image GL_ARB_invalidate_subdata GL_ARB_multi_bind GL_ARB_robustness GL_ARB_separate_shader_objects GL_ARB_texture_filter_anisotropic GL_ARB_texture_storage GL_ARB_texture_storage_multisample GL_ARB_vertex_array_object GL_KHR_debug Using driver workarounds: no-layout-qualifiers-on-old-glsl nv-zero-context-profile-mask nv-implementation-color-read-format-dsa-broken nv-cubemap-inconsistent-compressed-image-size nv-cubemap-broken-full-compressed-image-query nv-compressed-block-size-in-bits I0307 11:55:33.644829 6928 simulator.py:80] Loaded navmesh data/scene_datasets/gibson/Cantwell.navmesh 2023-03-07 11:55:33,645 initializing task Nav-v0 2023-03-07 11:55:33,651 Computing map for data/scene_datasets/gibson/Cantwell.glb Traceback (most recent call last): File "main.py", line 769, in main() File "main.py", line 273, in main local_map[:, 1, :, :], local_pose) File "/home/farjadnm/anaconda3/envs/habitat/lib/python3.7/site-packages/torch/nn/modules/module.py", line 547, in call result = self.forward(*input, kwargs) File "/home/farjadnm/Neural-SLAM/model.py", line 162, in forward resnet_output = self.resnet_l5(obs[:, :3, :, :]) File "/home/farjadnm/anaconda3/envs/habitat/lib/python3.7/site-packages/torch/nn/modules/module.py", line 547, in call result = self.forward(*input, *kwargs) File "/home/farjadnm/anaconda3/envs/habitat/lib/python3.7/site-packages/torch/nn/modules/container.py", line 92, in forward input = module(input) File "/home/farjadnm/anaconda3/envs/habitat/lib/python3.7/site-packages/torch/nn/modules/module.py", line 547, in call result = self.forward(input, kwargs) File "/home/farjadnm/anaconda3/envs/habitat/lib/python3.7/site-packages/torch/nn/modules/batchnorm.py", line 81, in forward exponential_average_factor, self.eps) File "/home/farjadnm/anaconda3/envs/habitat/lib/python3.7/site-packages/torch/nn/functional.py", line 1656, in batch_norm training, momentum, eps, torch.backends.cudnn.enabled RuntimeError: cuDNN error: CUDNN_STATUS_EXECUTION_FAILED Exception ignored in: <function VectorEnv.del at 0x7f8d90b5def0> Traceback (most recent call last): File "/home/farjadnm/Neural-SLAM/env/habitat/habitat_api/habitat/core/vector_env.py", line 487, in del self.close() File "/home/farjadnm/Neural-SLAM/env/habitat/habitat_api/habitat/core/vector_env.py", line 351, in close write_fn((CLOSE_COMMAND, None)) File "/home/farjadnm/anaconda3/envs/habitat/lib/python3.7/multiprocessing/connection.py", line 206, in send self._send_bytes(_ForkingPickler.dumps(obj)) File "/home/farjadnm/anaconda3/envs/habitat/lib/python3.7/multiprocessing/connection.py", line 404, in _send_bytes self._send(header + buf) File "/home/farjadnm/anaconda3/envs/habitat/lib/python3.7/multiprocessing/connection.py", line 368, in _send n = write(self._handle, buf) BrokenPipeError: [Errno 32] Broken pipe

How can I fix this issue? Thanks!

mylad13 commented 1 year ago

I started with a fresh Ubuntu 18.04, and carefully downloaded compatible pytorch and cuda versions for my graphics card. Now I get the following messages when running "python main.py -n1 --auto_gpu_config 0 --split val":

Dumping at ./tmp//models/exp1/ Namespace(alpha=0.99, auto_gpu_config=0, camera_height=1.25, clip_param=0.2, collision_threshold=0.2, cuda=True, du_scale=2, dump_location='./tmp/', entropy_coef=0.001, env_frame_height=256, env_frame_width=256, eps=1e-05, eval=0, exp_loss_coeff=1.0, exp_name='exp1', frame_height=128, frame_width=128, gamma=0.99, global_downscaling=2, global_hidden_size=256, global_lr=2.5e-05, goals_size=2, hfov=90.0, load_global='0', load_local='0', load_slam='0', local_hidden_size=512, local_optimizer='adam,lr=0.0001', local_policy_update_freq=5, log_interval=10, map_pred_threshold=0.5, map_resolution=5, map_size_cm=2400, max_episode_length=1000, max_grad_norm=0.5, no_cuda=False, noise_level=1.0, noisy_actions=1, noisy_odometry=1, num_episodes=1000000, num_global_steps=40, num_local_steps=25, num_mini_batch=0, num_processes=1, num_processes_on_first_gpu=0, num_processes_per_gpu=11, obs_threshold=1, obstacle_boundary=5, pose_loss_coeff=10000.0, ppo_epoch=4, pretrained_resnet=1, print_images=0, proj_loss_coeff=1.0, randomize_env_every=1000, save_interval=1, save_periodic=500000, save_trajectory_data='0', seed=1, short_goal_dist=1, sim_gpu_id=0, slam_batch_size=72, slam_iterations=10, slam_memory_size=500000, slam_optimizer='adam,lr=0.0001', split='val', task_config='tasks/pointnav_gibson.yaml', tau=0.95, total_num_scenes='auto', train_global=1, train_local=1, train_slam=1, use_deterministic_local=0, use_gae=False, use_pose_estimation=2, use_recurrent_global=0, use_recurrent_local=1, value_loss_coef=0.5, vis_type=1, vision_range=64, visualize=0) Loading data/scene_datasets/gibson/Cantwell.glb 2023-03-14 13:19:17,692 initializing sim Sim-v0 Renderer: NVIDIA GeForce RTX 3090/PCIe/SSE2 by NVIDIA Corporation OpenGL version: 4.6.0 NVIDIA 470.161.03 Using optional features: GL_ARB_ES2_compatibility GL_ARB_direct_state_access GL_ARB_get_texture_sub_image GL_ARB_invalidate_subdata GL_ARB_multi_bind GL_ARB_robustness GL_ARB_separate_shader_objects GL_ARB_texture_filter_anisotropic GL_ARB_texture_storage GL_ARB_texture_storage_multisample GL_ARB_vertex_array_object GL_KHR_debug Using driver workarounds: no-layout-qualifiers-on-old-glsl nv-zero-context-profile-mask nv-implementation-color-read-format-dsa-broken nv-cubemap-inconsistent-compressed-image-size nv-cubemap-broken-full-compressed-image-query nv-compressed-block-size-in-bits Trade::TinyGltfImporter::openData(): error opening file: JSON string too short. WARNING: Logging before InitGoogleLogging() is written to STDERR E0314 13:19:17.772595 355 ResourceManager.cpp:804] Cannot open file data/scene_datasets/gibson/Cantwell.glb E0314 13:19:17.772612 355 Simulator.cpp:106] cannot load data/scene_datasets/gibson/Cantwell.glb I0314 13:19:17.772698 355 SceneManager.h:24] Deconstructing SceneManager I0314 13:19:17.772702 355 SceneGraph.h:20] Deconstructing SceneGraph I0314 13:19:17.772709 355 Renderer.cpp:38] Deconstructing Renderer I0314 13:19:17.772725 355 WindowlessContext.cpp:240] Deconstructing GL context Process ForkServerProcess-1: Traceback (most recent call last): File "/home/farjadnm/anaconda3/envs/ANS/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap self.run() File "/home/farjadnm/anaconda3/envs/ANS/lib/python3.7/multiprocessing/process.py", line 99, in run self._target(*self._args, *self._kwargs) File "/home/farjadnm/Neural-SLAM/env/habitat/habitat_api/habitat/core/vector_env.py", line 148, in _worker_env env = env_fn(env_fn_args) File "/home/farjadnm/Neural-SLAM/env/habitat/init.py", line 22, in make_env_fn config_env=config_env, config_baseline=config_baseline, dataset=dataset File "/home/farjadnm/Neural-SLAM/env/habitat/exploration_env.py", line 83, in init super().init(config_env, dataset) File "/home/farjadnm/habitat-api/habitat/core/env.py", line 290, in init self._env = Env(config, dataset) File "/home/farjadnm/habitat-api/habitat/core/env.py", line 93, in init id_sim=self._config.SIMULATOR.TYPE, config=self._config.SIMULATOR File "/home/farjadnm/habitat-api/habitat/sims/registration.py", line 19, in make_sim return _sim(**kwargs) File "/home/farjadnm/habitat-api/habitat/sims/habitat_simulator/habitat_simulator.py", line 160, in init self._sim = habitat_sim.Simulator(self.sim_config) File "", line 7, in init self.attrs_post_init() File "/home/farjadnm/anaconda3/envs/ANS/lib/python3.7/site-packages/habitat_sim-0.1.2-py3.7-linux-x86_64.egg/habitat_sim/simulator.py", line 38, in attrs_post_init self.reconfigure(config) File "/home/farjadnm/anaconda3/envs/ANS/lib/python3.7/site-packages/habitat_sim-0.1.2-py3.7-linux-x86_64.egg/habitat_sim/simulator.py", line 104, in reconfigure self._config_backend(config) File "/home/farjadnm/anaconda3/envs/ANS/lib/python3.7/site-packages/habitat_sim-0.1.2-py3.7-linux-x86_64.egg/habitat_sim/simulator.py", line 56, in _config_backend self._sim = hsim.SimulatorBackend(config.sim_cfg) ValueError: Cannot load: data/scene_datasets/gibson/Cantwell.glb Traceback (most recent call last): File "main.py", line 769, in main() File "main.py", line 119, in main envs = make_vec_envs(args) File "/home/farjadnm/Neural-SLAM/env/init.py", line 7, in make_vec_envs envs = construct_envs(args) File "/home/farjadnm/Neural-SLAM/env/habitat/init.py", line 102, in construct_envs range(args.num_processes)) File "/home/farjadnm/Neural-SLAM/env/habitat/habitat_api/habitat/core/vector_env.py", line 117, in init read_fn() for read_fn in self._connection_read_fns File "/home/farjadnm/Neural-SLAM/env/habitat/habitat_api/habitat/core/vector_env.py", line 117, in read_fn() for read_fn in self._connection_read_fns File "/home/farjadnm/anaconda3/envs/ANS/lib/python3.7/multiprocessing/connection.py", line 250, in recv buf = self._recv_bytes() File "/home/farjadnm/anaconda3/envs/ANS/lib/python3.7/multiprocessing/connection.py", line 407, in _recv_bytes buf = self._recv(4) File "/home/farjadnm/anaconda3/envs/ANS/lib/python3.7/multiprocessing/connection.py", line 379, in _recv chunk = read(handle, remaining) ConnectionResetError: [Errno 104] Connection reset by peer Exception ignored in: <function VectorEnv.del at 0x7fac480fddd0> Traceback (most recent call last): File "/home/farjadnm/Neural-SLAM/env/habitat/habitat_api/habitat/core/vector_env.py", line 487, in del self.close() File "/home/farjadnm/Neural-SLAM/env/habitat/habitat_api/habitat/core/vector_env.py", line 351, in close write_fn((CLOSE_COMMAND, None)) File "/home/farjadnm/anaconda3/envs/ANS/lib/python3.7/multiprocessing/connection.py", line 206, in send self._send_bytes(_ForkingPickler.dumps(obj)) File "/home/farjadnm/anaconda3/envs/ANS/lib/python3.7/multiprocessing/connection.py", line 404, in _send_bytes self._send(header + buf) File "/home/farjadnm/anaconda3/envs/ANS/lib/python3.7/multiprocessing/connection.py", line 368, in _send n = write(self._handle, buf) BrokenPipeError: [Errno 32] Broken pipe

mylad13 commented 1 year ago

Fixed by replacing the contents of data/scene_datasets/gibson/ with files downloaded from here: https://github.com/devendrachaplot/Neural-SLAM/issues/59#issuecomment-1242411074 After that, I got this error in Neural-SLAM/utils/optimization.py: in getargspec raise ValueError("Function has keyword-only parameters or annotations" ValueError: Function has keyword-only parameters or annotations, use getfullargspec() API which can support them

I fixed that by replacing expected_args = inspect.getargspec(optim_fn.__init__)[0] with expected_args = inspect.getfullargspec(optim_fn.__init__)[0] in the optimization.py file.