search

isaac-sim / OmniIsaacGymEnvs

Reinforcement Learning Environments for Omniverse Isaac Gym
Other
764 stars 202 forks source link

FrankaDeformable is not working in 2023.1 #91

Open Skylark0924 opened 8 months ago

Skylark0924 commented 8 months ago

Bug

By running [PYTHON_PATH] scripts/rlgames_train.py task=FrankaDeformable, I got following error message:

...
Sim params does not have attribute:  physx
Sim params does not have attribute:  franka
Sim params does not have attribute:  beaker
Sim params does not have attribute:  cube
Pipeline:  GPU
Pipeline Device:  cuda:0
Sim Device:  GPU
Task Device: cuda:0
RL device:  cuda:0
2023-10-27 15:24:44 [21,326ms] [Warning] [omni.hydra.scene_delegate.plugin] Calling getBypassRenderSkelMeshProcessing for prim /World/envs/env_0/franka/panda_hand/visuals.proto_visuals_id0 that has not been populated
~*~*~*
~*~*~* Direct GPU Helper:
~*~*~*   1024 rigid dynamics
~*~*~*   1024 articulations, maxLinks=11
~*~*~*
2023-10-27 15:24:52 [29,322ms] [Error] [omni.physx.tensors.plugin] CUDA error: an illegal memory access was encountered: ../../../extensions/runtime/source/omni.physx.tensors/plugins/gpu/GpuSoftBodyView.cpp: 277
2023-10-27 15:24:52 [29,322ms] [Error] [omni.physx.tensors.plugin] CUDA error: an illegal memory access was encountered: ../../../extensions/runtime/source/omni.physx.tensors/plugins/gpu/GpuSoftBodyView.cpp: 274
2023-10-27 15:24:52 [29,322ms] [Error] [omni.physx.tensors.plugin] CUDA error: an illegal memory access was encountered: ../../../extensions/runtime/source/omni.physx.tensors/plugins/gpu/GpuSoftBodyView.cpp: 277
2023-10-27 15:24:52 [29,322ms] [Error] [omni.physx.tensors.plugin] CUDA error: an illegal memory access was encountered: ../../../extensions/runtime/source/omni.physx.tensors/plugins/gpu/GpuSoftBodyView.cpp: 274
2023-10-27 15:24:52 [29,322ms] [Error] [omni.physx.tensors.plugin] CUDA error: an illegal memory access was encountered: ../../../extensions/runtime/source/omni.physx.tensors/plugins/gpu/GpuSoftBodyView.cpp: 277
2023-10-27 15:24:52 [29,352ms] [Error] [omni.physx.tensors.plugin] CUDA error: an illegal memory access was encountered: ../../../extensions/runtime/source/omni.physx.tensors/plugins/gpu/GpuArticulationView.cpp: 71
2023-10-27 15:24:52 [29,374ms] [Error] [omni.kit.app._impl] [py stderr]: Error executing job with overrides: ['task=FrankaDeformable', 'test=True', 'checkpoint=omniverse://localhost/NVIDIA/Assets/Isaac/2023.1.0/Isaac/Samples/OmniIsaacGymEnvs/Checkpoints/franka_deformable.pth']

Error executing job with overrides: ['task=FrankaDeformable', 'test=True', 'checkpoint=omniverse://localhost/NVIDIA/Assets/Isaac/2023.1.0/Isaac/Samples/OmniIsaacGymEnvs/Checkpoints/franka_deformable.pth']
2023-10-27 15:24:52 [29,374ms] [Error] [omni.kit.app._impl] [py stderr]: Traceback (most recent call last):

Traceback (most recent call last):
2023-10-27 15:24:52 [29,374ms] [Error] [omni.kit.app._impl] [py stderr]:   File "/home/skylark/Github/OmniIsaacGymEnvs/omniisaacgymenvs/scripts/rlgames_train.py", line 150, in <module>

  File "/home/skylark/Github/OmniIsaacGymEnvs/omniisaacgymenvs/scripts/rlgames_train.py", line 150, in <module>
2023-10-27 15:24:52 [29,374ms] [Error] [omni.kit.app._impl] [py stderr]:     
2023-10-27 15:24:52 [29,374ms] [Error] [omni.kit.app._impl] [py stderr]: parse_hydra_configs()
2023-10-27 15:24:52 [29,374ms] [Error] [omni.kit.app._impl] [py stderr]: 

    parse_hydra_configs()
2023-10-27 15:24:52 [29,374ms] [Error] [omni.kit.app._impl] [py stderr]:   File "/home/skylark/.local/share/ov/pkg/isaac_sim-2023.1.0/kit/python/lib/python3.10/site-packages/hydra/main.py", line 94, in decorated_main

  File "/home/skylark/.local/share/ov/pkg/isaac_sim-2023.1.0/kit/python/lib/python3.10/site-packages/hydra/main.py", line 94, in decorated_main
2023-10-27 15:24:52 [29,374ms] [Error] [omni.kit.app._impl] [py stderr]:     
2023-10-27 15:24:52 [29,374ms] [Error] [omni.kit.app._impl] [py stderr]: _run_hydra(
2023-10-27 15:24:52 [29,374ms] [Error] [omni.kit.app._impl] [py stderr]: 

    _run_hydra(
2023-10-27 15:24:52 [29,374ms] [Error] [omni.kit.app._impl] [py stderr]:   File "/home/skylark/.local/share/ov/pkg/isaac_sim-2023.1.0/kit/python/lib/python3.10/site-packages/hydra/_internal/utils.py", line 394, in _run_hydra

  File "/home/skylark/.local/share/ov/pkg/isaac_sim-2023.1.0/kit/python/lib/python3.10/site-packages/hydra/_internal/utils.py", line 394, in _run_hydra
2023-10-27 15:24:52 [29,374ms] [Error] [omni.kit.app._impl] [py stderr]:     
2023-10-27 15:24:52 [29,374ms] [Error] [omni.kit.app._impl] [py stderr]: _run_app(
2023-10-27 15:24:52 [29,374ms] [Error] [omni.kit.app._impl] [py stderr]: 

    _run_app(
2023-10-27 15:24:52 [29,374ms] [Error] [omni.kit.app._impl] [py stderr]:   File "/home/skylark/.local/share/ov/pkg/isaac_sim-2023.1.0/kit/python/lib/python3.10/site-packages/hydra/_internal/utils.py", line 457, in _run_app

  File "/home/skylark/.local/share/ov/pkg/isaac_sim-2023.1.0/kit/python/lib/python3.10/site-packages/hydra/_internal/utils.py", line 457, in _run_app
2023-10-27 15:24:52 [29,374ms] [Error] [omni.kit.app._impl] [py stderr]:     
2023-10-27 15:24:52 [29,374ms] [Error] [omni.kit.app._impl] [py stderr]: run_and_report(
2023-10-27 15:24:52 [29,375ms] [Error] [omni.kit.app._impl] [py stderr]: 

    run_and_report(
2023-10-27 15:24:52 [29,375ms] [Error] [omni.kit.app._impl] [py stderr]:   File "/home/skylark/.local/share/ov/pkg/isaac_sim-2023.1.0/kit/python/lib/python3.10/site-packages/hydra/_internal/utils.py", line 223, in run_and_report

  File "/home/skylark/.local/share/ov/pkg/isaac_sim-2023.1.0/kit/python/lib/python3.10/site-packages/hydra/_internal/utils.py", line 223, in run_and_report
2023-10-27 15:24:52 [29,375ms] [Error] [omni.kit.app._impl] [py stderr]:     
2023-10-27 15:24:52 [29,375ms] [Error] [omni.kit.app._impl] [py stderr]: raise ex
2023-10-27 15:24:52 [29,375ms] [Error] [omni.kit.app._impl] [py stderr]: 

    raise ex
2023-10-27 15:24:52 [29,375ms] [Error] [omni.kit.app._impl] [py stderr]:   File "/home/skylark/.local/share/ov/pkg/isaac_sim-2023.1.0/kit/python/lib/python3.10/site-packages/hydra/_internal/utils.py", line 220, in run_and_report

  File "/home/skylark/.local/share/ov/pkg/isaac_sim-2023.1.0/kit/python/lib/python3.10/site-packages/hydra/_internal/utils.py", line 220, in run_and_report
2023-10-27 15:24:52 [29,375ms] [Error] [omni.kit.app._impl] [py stderr]:     
2023-10-27 15:24:52 [29,375ms] [Error] [omni.kit.app._impl] [py stderr]: return func()
2023-10-27 15:24:52 [29,375ms] [Error] [omni.kit.app._impl] [py stderr]: 

    return func()
2023-10-27 15:24:52 [29,375ms] [Error] [omni.kit.app._impl] [py stderr]:   File "/home/skylark/.local/share/ov/pkg/isaac_sim-2023.1.0/kit/python/lib/python3.10/site-packages/hydra/_internal/utils.py", line 458, in <lambda>

  File "/home/skylark/.local/share/ov/pkg/isaac_sim-2023.1.0/kit/python/lib/python3.10/site-packages/hydra/_internal/utils.py", line 458, in <lambda>
2023-10-27 15:24:52 [29,375ms] [Error] [omni.kit.app._impl] [py stderr]:     
2023-10-27 15:24:52 [29,375ms] [Error] [omni.kit.app._impl] [py stderr]: lambda: hydra.run(
2023-10-27 15:24:52 [29,375ms] [Error] [omni.kit.app._impl] [py stderr]: 

    lambda: hydra.run(
2023-10-27 15:24:52 [29,375ms] [Error] [omni.kit.app._impl] [py stderr]:   File "/home/skylark/.local/share/ov/pkg/isaac_sim-2023.1.0/kit/python/lib/python3.10/site-packages/hydra/_internal/hydra.py", line 132, in run

  File "/home/skylark/.local/share/ov/pkg/isaac_sim-2023.1.0/kit/python/lib/python3.10/site-packages/hydra/_internal/hydra.py", line 132, in run
2023-10-27 15:24:52 [29,375ms] [Error] [omni.kit.app._impl] [py stderr]:     
2023-10-27 15:24:52 [29,375ms] [Error] [omni.kit.app._impl] [py stderr]: _ = ret.return_value
2023-10-27 15:24:52 [29,375ms] [Error] [omni.kit.app._impl] [py stderr]: 

    _ = ret.return_value
2023-10-27 15:24:52 [29,375ms] [Error] [omni.kit.app._impl] [py stderr]:   File "/home/skylark/.local/share/ov/pkg/isaac_sim-2023.1.0/kit/python/lib/python3.10/site-packages/hydra/core/utils.py", line 260, in return_value

  File "/home/skylark/.local/share/ov/pkg/isaac_sim-2023.1.0/kit/python/lib/python3.10/site-packages/hydra/core/utils.py", line 260, in return_value
2023-10-27 15:24:52 [29,375ms] [Error] [omni.kit.app._impl] [py stderr]:     
2023-10-27 15:24:52 [29,375ms] [Error] [omni.kit.app._impl] [py stderr]: raise self._return_value
2023-10-27 15:24:52 [29,375ms] [Error] [omni.kit.app._impl] [py stderr]: 

    raise self._return_value
2023-10-27 15:24:52 [29,375ms] [Error] [omni.kit.app._impl] [py stderr]:   File "/home/skylark/.local/share/ov/pkg/isaac_sim-2023.1.0/kit/python/lib/python3.10/site-packages/hydra/core/utils.py", line 186, in run_job

  File "/home/skylark/.local/share/ov/pkg/isaac_sim-2023.1.0/kit/python/lib/python3.10/site-packages/hydra/core/utils.py", line 186, in run_job
2023-10-27 15:24:52 [29,376ms] [Error] [omni.kit.app._impl] [py stderr]:     
2023-10-27 15:24:52 [29,376ms] [Error] [omni.kit.app._impl] [py stderr]: ret.return_value = task_function(task_cfg)
2023-10-27 15:24:52 [29,376ms] [Error] [omni.kit.app._impl] [py stderr]: 

    ret.return_value = task_function(task_cfg)
2023-10-27 15:24:52 [29,376ms] [Error] [omni.kit.app._impl] [py stderr]:   File "/home/skylark/Github/OmniIsaacGymEnvs/omniisaacgymenvs/scripts/rlgames_train.py", line 121, in parse_hydra_configs

  File "/home/skylark/Github/OmniIsaacGymEnvs/omniisaacgymenvs/scripts/rlgames_train.py", line 121, in parse_hydra_configs
2023-10-27 15:24:52 [29,376ms] [Error] [omni.kit.app._impl] [py stderr]:     
2023-10-27 15:24:52 [29,376ms] [Error] [omni.kit.app._impl] [py stderr]: task = initialize_task(cfg_dict, env)
2023-10-27 15:24:52 [29,376ms] [Error] [omni.kit.app._impl] [py stderr]: 

    task = initialize_task(cfg_dict, env)
2023-10-27 15:24:52 [29,376ms] [Error] [omni.kit.app._impl] [py stderr]:   File "/home/skylark/Github/OmniIsaacGymEnvs/omniisaacgymenvs/utils/task_util.py", line 103, in initialize_task

  File "/home/skylark/Github/OmniIsaacGymEnvs/omniisaacgymenvs/utils/task_util.py", line 103, in initialize_task
2023-10-27 15:24:52 [29,376ms] [Error] [omni.kit.app._impl] [py stderr]:     
2023-10-27 15:24:52 [29,376ms] [Error] [omni.kit.app._impl] [py stderr]: env.set_task(
2023-10-27 15:24:52 [29,376ms] [Error] [omni.kit.app._impl] [py stderr]: 

    env.set_task(
2023-10-27 15:24:52 [29,376ms] [Error] [omni.kit.app._impl] [py stderr]:   File "/home/skylark/Github/OmniIsaacGymEnvs/omniisaacgymenvs/envs/vec_env_rlgames.py", line 47, in set_task

  File "/home/skylark/Github/OmniIsaacGymEnvs/omniisaacgymenvs/envs/vec_env_rlgames.py", line 47, in set_task
2023-10-27 15:24:52 [29,376ms] [Error] [omni.kit.app._impl] [py stderr]:     
2023-10-27 15:24:52 [29,376ms] [Error] [omni.kit.app._impl] [py stderr]: super().set_task(task, backend, sim_params, init_sim, rendering_dt)
2023-10-27 15:24:52 [29,376ms] [Error] [omni.kit.app._impl] [py stderr]: 

    super().set_task(task, backend, sim_params, init_sim, rendering_dt)
2023-10-27 15:24:52 [29,376ms] [Error] [omni.kit.app._impl] [py stderr]:   File "/home/skylark/.local/share/ov/pkg/isaac_sim-2023.1.0/exts/omni.isaac.gym/omni/isaac/gym/vec_env/vec_env_base.py", line 126, in set_task

  File "/home/skylark/.local/share/ov/pkg/isaac_sim-2023.1.0/exts/omni.isaac.gym/omni/isaac/gym/vec_env/vec_env_base.py", line 126, in set_task
2023-10-27 15:24:52 [29,376ms] [Error] [omni.kit.app._impl] [py stderr]:     
2023-10-27 15:24:52 [29,376ms] [Error] [omni.kit.app._impl] [py stderr]: self._world.reset()
2023-10-27 15:24:52 [29,376ms] [Error] [omni.kit.app._impl] [py stderr]: 

    self._world.reset()
2023-10-27 15:24:52 [29,376ms] [Error] [omni.kit.app._impl] [py stderr]:   File "/home/skylark/.local/share/ov/pkg/isaac_sim-2023.1.0/exts/omni.isaac.core/omni/isaac/core/world/world.py", line 283, in reset

  File "/home/skylark/.local/share/ov/pkg/isaac_sim-2023.1.0/exts/omni.isaac.core/omni/isaac/core/world/world.py", line 283, in reset
2023-10-27 15:24:52 [29,376ms] [Error] [omni.kit.app._impl] [py stderr]:     
2023-10-27 15:24:52 [29,376ms] [Error] [omni.kit.app._impl] [py stderr]: self._scene._finalize(self.physics_sim_view)
2023-10-27 15:24:52 [29,376ms] [Error] [omni.kit.app._impl] [py stderr]: 

    self._scene._finalize(self.physics_sim_view)
2023-10-27 15:24:52 [29,376ms] [Error] [omni.kit.app._impl] [py stderr]:   File "/home/skylark/.local/share/ov/pkg/isaac_sim-2023.1.0/exts/omni.isaac.core/omni/isaac/core/scenes/scene.py", line 320, in _finalize

  File "/home/skylark/.local/share/ov/pkg/isaac_sim-2023.1.0/exts/omni.isaac.core/omni/isaac/core/scenes/scene.py", line 320, in _finalize
2023-10-27 15:24:52 [29,377ms] [Error] [omni.kit.app._impl] [py stderr]:     
2023-10-27 15:24:52 [29,377ms] [Error] [omni.kit.app._impl] [py stderr]: articulated_view.initialize(physics_sim_view)
2023-10-27 15:24:52 [29,377ms] [Error] [omni.kit.app._impl] [py stderr]: 

    articulated_view.initialize(physics_sim_view)
2023-10-27 15:24:52 [29,377ms] [Error] [omni.kit.app._impl] [py stderr]:   File "/home/skylark/Github/OmniIsaacGymEnvs/omniisaacgymenvs/robots/articulations/views/franka_view.py", line 30, in initialize

  File "/home/skylark/Github/OmniIsaacGymEnvs/omniisaacgymenvs/robots/articulations/views/franka_view.py", line 30, in initialize
2023-10-27 15:24:52 [29,377ms] [Error] [omni.kit.app._impl] [py stderr]:     
2023-10-27 15:24:52 [29,377ms] [Error] [omni.kit.app._impl] [py stderr]: super().initialize(physics_sim_view)
2023-10-27 15:24:52 [29,377ms] [Error] [omni.kit.app._impl] [py stderr]: 

    super().initialize(physics_sim_view)
2023-10-27 15:24:52 [29,377ms] [Error] [omni.kit.app._impl] [py stderr]:   File "/home/skylark/.local/share/ov/pkg/isaac_sim-2023.1.0/exts/omni.isaac.core/omni/isaac/core/articulations/articulation_view.py", line 221, in initialize

  File "/home/skylark/.local/share/ov/pkg/isaac_sim-2023.1.0/exts/omni.isaac.core/omni/isaac/core/articulations/articulation_view.py", line 221, in initialize
2023-10-27 15:24:52 [29,377ms] [Error] [omni.kit.app._impl] [py stderr]:     
2023-10-27 15:24:52 [29,377ms] [Error] [omni.kit.app._impl] [py stderr]: self._default_kps, self._default_kds = self.get_gains(clone=True)
2023-10-27 15:24:52 [29,377ms] [Error] [omni.kit.app._impl] [py stderr]: 

    self._default_kps, self._default_kds = self.get_gains(clone=True)
2023-10-27 15:24:52 [29,377ms] [Error] [omni.kit.app._impl] [py stderr]:   File "/home/skylark/.local/share/ov/pkg/isaac_sim-2023.1.0/exts/omni.isaac.core/omni/isaac/core/articulations/articulation_view.py", line 1807, in get_gains

  File "/home/skylark/.local/share/ov/pkg/isaac_sim-2023.1.0/exts/omni.isaac.core/omni/isaac/core/articulations/articulation_view.py", line 1807, in get_gains
2023-10-27 15:24:52 [29,377ms] [Error] [omni.kit.app._impl] [py stderr]:     
2023-10-27 15:24:52 [29,377ms] [Error] [omni.kit.app._impl] [py stderr]: indices = self._backend_utils.resolve_indices(indices, self.count, device=self._device)
2023-10-27 15:24:52 [29,377ms] [Error] [omni.kit.app._impl] [py stderr]: 

    indices = self._backend_utils.resolve_indices(indices, self.count, device=self._device)
2023-10-27 15:24:52 [29,377ms] [Error] [omni.kit.app._impl] [py stderr]:   File "/home/skylark/.local/share/ov/pkg/isaac_sim-2023.1.0/exts/omni.isaac.core/omni/isaac/core/utils/torch/tensor.py", line 54, in resolve_indices

  File "/home/skylark/.local/share/ov/pkg/isaac_sim-2023.1.0/exts/omni.isaac.core/omni/isaac/core/utils/torch/tensor.py", line 54, in resolve_indices
2023-10-27 15:24:52 [29,377ms] [Error] [omni.kit.app._impl] [py stderr]:     
2023-10-27 15:24:52 [29,377ms] [Error] [omni.kit.app._impl] [py stderr]: result = torch.arange(count, device=device)
2023-10-27 15:24:52 [29,377ms] [Error] [omni.kit.app._impl] [py stderr]: 

    result = torch.arange(count, device=device)
2023-10-27 15:24:52 [29,377ms] [Error] [omni.kit.app._impl] [py stderr]: RuntimeError
2023-10-27 15:24:52 [29,377ms] [Error] [omni.kit.app._impl] [py stderr]: : 
2023-10-27 15:24:52 [29,378ms] [Error] [omni.kit.app._impl] [py stderr]: CUDA error: an illegal memory access was encountered
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

RuntimeError: CUDA error: an illegal memory access was encountered
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
2023-10-27 15:24:52 [29,378ms] [Error] [omni.kit.app._impl] [py stderr]: 

2023-10-27 15:24:52 [29,381ms] [Warning] [carb] [Plugin: omni.spectree.delegate.plugin] Module /home/skylark/.local/share/ov/pkg/isaac_sim-2023.1.0/kit/exts/omni.usd_resolver/bin/libomni.spectree.delegate.plugin.so remained loaded after unload request
2023-10-27 15:24:52 [29,383ms] [Warning] [omni.stageupdate.plugin] Deprecated: direct use of IStageUpdate callbacks is deprecated. Use IStageUpdate::getStageUpdate instead.
2023-10-27 15:24:52 [29,384ms] [Error] [omni.physx.fabric.plugin] CUDA error: an illegal memory access was encountered: ../../../extensions/runtime/source/omni.physx.fabric/plugins/DirectGpuHelper.cpp: 349
2023-10-27 15:24:52 [29,384ms] [Error] [omni.physx.fabric.plugin] CUDA error: an illegal memory access was encountered: ../../../extensions/runtime/source/omni.physx.fabric/plugins/DirectGpuHelper.cpp: 352
2023-10-27 15:24:52 [29,384ms] [Error] [omni.physx.fabric.plugin] CUDA error: an illegal memory access was encountered: ../../../extensions/runtime/source/omni.physx.fabric/plugins/DirectGpuHelper.cpp: 355
2023-10-27 15:24:52 [29,384ms] [Error] [omni.physx.fabric.plugin] CUDA error: an illegal memory access was encountered: ../../../extensions/runtime/source/omni.physx.fabric/plugins/DirectGpuHelper.cpp: 358
2023-10-27 15:24:52 [29,384ms] [Error] [omni.physx.fabric.plugin] CUDA error: an illegal memory access was encountered: ../../../extensions/runtime/source/omni.physx.fabric/plugins/DirectGpuHelper.cpp: 361
2023-10-27 15:24:52 [29,388ms] [Warning] [carb.audio.context] 1 contexts were leaked
2023-10-27 15:24:52 [29,433ms] [Warning] [carb] Recursive unloadAllPlugins() detected!
2023-10-27 15:24:52 [29,442ms] [Warning] [omni.core.ITypeFactory] Module /home/skylark/.local/share/ov/pkg/isaac_sim-2023.1.0/kit/exts/omni.activity.core/bin/libomni.activity.core.plugin.so remained loaded after unload request.

Trails

  1. Other tasks (like FrankaCabinet) work well in 2023.1. So I think it should be an issue about the deformable simulation.
  2. Since the error shows [Error] [omni.physx.fabric.plugin], I have tried to launch isaac sim to open and close these extensions, but nothing changed. image
kellyguo11 commented 8 months ago

Hi there, can you try lowering the number of environments with the num_envs flag? It's possible the machine ran out of memory during the deformable simulation.

Skylark0924 commented 8 months ago

Thanks for your reply. I have tried to modify the num_envs to 64/32/16, the error keeps the same

dhruvkm2402 commented 8 months ago

Hi @Skylark0924 , It worked for me with the default num_envs settings. I have a workstation though with RTX A6000 and 256GB RAM

XEL-Maker commented 7 months ago

Hi @kellyguo11

Hi there, can you try lowering the number of environments with the num_envs flag? It's possible the machine ran out of memory during the deformable simulation.

I tried above solution but it's not worked well.(Laptop with RTX 4070 and 32GB RAM, num_envs: 4 and 8 tried) The env is well setted and after it crash. It also happens at extension workflow.(clicking start after loading env) I treid it wih default setting at my workstation though with RTX A4000 and 32GB RAM it worked. Both are Window 11 OS. Including log. kit_20231114_092711.log