isaac-sim / IsaacLab

Unified framework for robot learning built on NVIDIA Isaac Sim
https://isaac-sim.github.io/IsaacLab
Other
2.28k stars 934 forks source link

[Question] Encountering Unknown Error: [Error] [omni.physx.cooking.plugin] opRes == OperationResult::SUCCESS is false. #1409

Open H-Hisamichi opened 1 week ago

H-Hisamichi commented 1 week ago

Hello everyone,

I encountered an unknown error while training my original Hexapod using reinforcement learning.

First, I get this warning: [Warning] [omni.ujitso] UJITSO: Build storage validation failed for Storage status: 1, and then the error [Error] [omni.physx.cooking.plugin] opRes == OperationResult::SUCCESS is false. appears. The error occurred after the log [INFO]: Starting the simulation. This may take a few seconds. Please wait....

Sometimes this error causes a buffer error, but other times the training continues without any issues. I couldn't find any information about this error. Does anyone know what this means?

Thank you!

RandomOakForest commented 1 week ago

Could you please confirm the version of Isaac Lab you are using? Have you tried this in the 1.2 release? Thanks!

H-Hisamichi commented 1 week ago

Hello @RandomOakForest ,

I'm using Isaac Lab v1.2.0 and Isaac Sim v4.2.0. Thanks again!

RandomOakForest commented 1 week ago

Thank you for submitting this. We have take note of this error. If it is causing a crash on your side, please submit a bug report. Thanks.

H-Hisamichi commented 6 days ago

Hello @RandomOakForest,

Thank you for your response. The current error code does not cause a crash. However, I am not sure if this error affects policy. I will provide more information about my environment. Isaac Sim: 4.2.0 Isaac Lab: 1.2.0 RL code: based on anymal_c(direct), modified for original hexapod.

The display of error codes has also been changed. The cause is unknown. Example of error code pattern 1:

[INFO] Using python from: /home/robot/anaconda3/envs/iccar2025/bin/python
[INFO][AppLauncher]: Loading experience file: /home/robot/IsaacLab_v1.2.0/IsaacLab/source/apps/isaaclab.python.headless.rendering.kit
Loading user config located at: '/home/robot/.local/share/ov/pkg/isaac-sim-4.2.0/kit/data/Kit/Isaac-Sim/4.2/user.config.json'
[Info] [carb] Logging to file: /home/robot/.local/share/ov/pkg/isaac-sim-4.2.0/kit/logs/Kit/Isaac-Sim/4.2/kit_20241117_101409.log
2024-11-17 01:14:10 [0ms] [Warning] [omni.kit.app.plugin] No crash reporter present, dumps uploading isn't available.
2024-11-17 01:14:10 [673ms] [Warning] [carb.windowing-glfw.plugin] GLFW initialization failed.
2024-11-17 01:14:10 [673ms] [Warning] [carb] Failed to startup plugin carb.windowing-glfw.plugin (interfaces: [carb::windowing::IGLContext v1.0],[carb::windowing::IWindowing v1.4]) (impl: carb.windowing-glfw.plugin)
2024-11-17 01:14:10 [701ms] [Warning] [omni.platforminfo.plugin] failed to open the default display.  Can't verify X Server version.

|---------------------------------------------------------------------------------------------|
| Driver Version: 535.183.01    | Graphics API: Vulkan
|=============================================================================================|
| GPU | Name                             | Active | LDA | GPU Memory | Vendor-ID | LUID       |
|     |                                  |        |     |            | Device-ID | UUID       |
|     |                                  |        |     |            | Bus-ID    |            |
|---------------------------------------------------------------------------------------------|
| 0   | NVIDIA RTX A4000                 | Yes: 0 |     | 16622   MB | 10de      | 0          |
|     |                                  |        |     |            | 24b0      | 44ccd3c0.. |
|     |                                  |        |     |            | 6         |            |
|=============================================================================================|
| OS: 22.04.5 LTS (Jammy Jellyfish) ubuntu, Version: 22.04.5, Kernel: 6.8.0-48-generic
| Processor: AMD Ryzen 7 2700 Eight-Core Processor | Cores: 8 | Logical: 16
|---------------------------------------------------------------------------------------------|
| Total Memory (MB): 48078 | Free Memory: 41052
| Total Page/Swap (MB): 2047 | Free Page/Swap: 2047
|---------------------------------------------------------------------------------------------|
2024-11-17 01:14:11 [1,319ms] [Warning] [gpu.foundation.plugin] IOMMU is enabled.
2024-11-17 01:14:12 [2,183ms] [Warning] [carb.windowing-glfw.plugin] GLFW initialization failed.
2024-11-17 01:14:12 [2,183ms] [Warning] [carb] Failed to startup plugin carb.windowing-glfw.plugin (interfaces: [carb::windowing::IGLContext v1.0],[carb::windowing::IWindowing v1.4]) (impl: carb.windowing-glfw.plugin)
2024-11-17 01:14:12 [2,193ms] [Warning] [carb.windowing-glfw.plugin] GLFW initialization failed.
2024-11-17 01:14:12 [2,193ms] [Warning] [carb] Failed to startup plugin carb.windowing-glfw.plugin (interfaces: [carb::windowing::IGLContext v1.0],[carb::windowing::IWindowing v1.4]) (impl: carb.windowing-glfw.plugin)
2024-11-17 01:14:12 [2,201ms] [Warning] [carb.windowing-glfw.plugin] GLFW initialization failed.
2024-11-17 01:14:12 [2,201ms] [Warning] [carb] Failed to startup plugin carb.windowing-glfw.plugin (interfaces: [carb::windowing::IGLContext v1.0],[carb::windowing::IWindowing v1.4]) (impl: carb.windowing-glfw.plugin)
2024-11-17 01:14:12 [2,210ms] [Warning] [carb.windowing-glfw.plugin] GLFW initialization failed.
2024-11-17 01:14:12 [2,210ms] [Warning] [carb] Failed to startup plugin carb.windowing-glfw.plugin (interfaces: [carb::windowing::IGLContext v1.0],[carb::windowing::IWindowing v1.4]) (impl: carb.windowing-glfw.plugin)
2024-11-17 01:14:12 [2,212ms] [Warning] [carb.windowing-glfw.plugin] GLFW initialization failed.
2024-11-17 01:14:12 [2,212ms] [Warning] [carb] Failed to startup plugin carb.windowing-glfw.plugin (interfaces: [carb::windowing::IGLContext v1.0],[carb::windowing::IWindowing v1.4]) (impl: carb.windowing-glfw.plugin)
2024-11-17 01:14:12 [2,213ms] [Warning] [carb.windowing-glfw.plugin] GLFW initialization failed.
2024-11-17 01:14:12 [2,213ms] [Warning] [carb] Failed to startup plugin carb.windowing-glfw.plugin (interfaces: [carb::windowing::IGLContext v1.0],[carb::windowing::IWindowing v1.4]) (impl: carb.windowing-glfw.plugin)
2024-11-17 01:14:15 [5,363ms] [Warning] [omni.replicator.core.scripts.annotators] Annotator PostProcessDispatch is already registered, overwriting annotator template
formatted_tuple: (0.5, 0.5, 0.5, 0.5)
[INFO]: Parsing configuration from: omni.isaac.lab_tasks.direct.AT3R.AT3R_rolling_env:AT3RRoughEnvCfg
[INFO]: Parsing configuration from: /home/robot/IsaacLab_v1.2.0/IsaacLab/source/extensions/omni.isaac.lab_tasks/omni/isaac/lab_tasks/direct/AT3R/agents/rl_games_rough_ppo_cfg.yaml
[INFO] Logging experiment in directory: /home/robot/IsaacLab_v1.2.0/IsaacLab/logs/rl_games/AT3R_rough_direct
Setting seed: 42
[INFO]: Base environment:
    Environment device    : cuda:0
    Environment seed      : 42
    Physics step-size     : 0.005
    Rendering step-size   : 0.02
    Environment step-size : 0.02
[INFO] Generating terrains based on curriculum took : 1.656922 seconds
[INFO]: Time taken for scene creation : 4.838669 seconds
[INFO]: Scene manager:  <class InteractiveScene>
    Number of environments: 1024
    Environment spacing   : 4.0
    Source prim name      : /World/envs/env_0
    Global prim paths     : ['/World/ground']
    Replicate physics     : True
[INFO]: Starting the simulation. This may take a few seconds. Please wait...
2024-11-17 01:14:30 [20,747ms] [Warning] [omni.ujitso] UJITSO : Build storage validation failed for  Storage status: 1
2024-11-17 01:14:30 [20,747ms] [Error] [omni.physx.cooking.plugin] opRes == OperationResult::SUCCESS is false.
2024-11-17 01:14:30 [20,749ms] [Warning] [omni.ujitso] UJITSO : Build storage validation failed for  Storage status: 1
2024-11-17 01:14:30 [20,749ms] [Error] [omni.physx.cooking.plugin] opRes == OperationResult::SUCCESS is false.
[INFO]: Time taken for simulation start : 57.431758 seconds
[INFO] Event Manager:  <EventManager> contains 1 active terms.
+--------------------------------------+
| Active Event Terms in Mode: 'startup' |
+----------+---------------------------+
|  Index   | Name                      |
+----------+---------------------------+
|    0     | physics_material          |
|    1     | add_base_mass             |
+----------+---------------------------+

[INFO]: Completed setting up the environment...
[INFO] Recording videos during training.
        video_folder: /home/robot/IsaacLab_v1.2.0/IsaacLab/logs/rl_games/AT3R_rough_direct/2024-11-17_10-14-20/videos/train
        step_trigger: lambda step: step % args_cli.video_interval == 0
        video_length: 400
        disable_logger: True
/home/robot/anaconda3/envs/iccar2025/lib/python3.10/site-packages/gymnasium/core.py:297: UserWarning: WARN: env.is_vector_env to get variables from other wrappers is deprecated and will be removed in v1.0, to get this variable you can do `env.unwrapped.is_vector_env` for environment variables or `env.get_attr('is_vector_env')` that will search the reminding wrappers.
  logger.warn(
self.seed = 42
Started to train
Exact experiment name requested from command line: 2024-11-17_10-14-20
seq_length: 4
current training device: cuda:0
/home/robot/anaconda3/envs/iccar2025/lib/python3.10/site-packages/rl_games/common/a2c_common.py:254: FutureWarning: `torch.cuda.amp.GradScaler(args...)` is deprecated. Please use `torch.amp.GradScaler('cuda', args...)` instead.
  self.scaler = torch.cuda.amp.GradScaler(enabled=self.mixed_precision)
build mlp: 252
RunningMeanStd:  (1,)
2024-11-17 01:15:27 [77,183ms] [Warning] [omni.syntheticdata.plugin] OgnSdPostRenderVarToHost : rendervar copy from texture directly to host buffer is counter-performant. Please use copy from texture to device buffer first.
/home/robot/anaconda3/envs/iccar2025/lib/python3.10/site-packages/rl_games/algos_torch/a2c_continuous.py:106: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  with torch.cuda.amp.autocast(enabled=self.mixed_precision):
fps step: 4422 fps step and policy inference: 4368 fps total: 4190 epoch: 1/5000 frames: 0
fps step: 6197 fps step and policy inference: 6106 fps total: 6055 epoch: 2/5000 frames: 24576

Example of error code pattern 2:

[INFO] Using python from: /home/robot/anaconda3/envs/iccar2025/bin/python
[INFO][AppLauncher]: Loading experience file: /home/robot/IsaacLab_v1.2.0/IsaacLab/source/apps/isaaclab.python.headless.rendering.kit
Loading user config located at: '/home/robot/.local/share/ov/pkg/isaac-sim-4.2.0/kit/data/Kit/Isaac-Sim/4.2/user.config.json'
[Info] [carb] Logging to file: /home/robot/.local/share/ov/pkg/isaac-sim-4.2.0/kit/logs/Kit/Isaac-Sim/4.2/kit_20241117_123841.log
2024-11-17 03:38:41 [0ms] [Warning] [omni.kit.app.plugin] No crash reporter present, dumps uploading isn't available.
2024-11-17 03:38:42 [744ms] [Warning] [carb.windowing-glfw.plugin] GLFW initialization failed.
2024-11-17 03:38:42 [744ms] [Warning] [carb] Failed to startup plugin carb.windowing-glfw.plugin (interfaces: [carb::windowing::IGLContext v1.0],[carb::windowing::IWindowing v1.4]) (impl: carb.windowing-glfw.plugin)
2024-11-17 03:38:42 [770ms] [Warning] [omni.platforminfo.plugin] failed to open the default display.  Can't verify X Server version.

|---------------------------------------------------------------------------------------------|
| Driver Version: 535.183.01    | Graphics API: Vulkan
|=============================================================================================|
| GPU | Name                             | Active | LDA | GPU Memory | Vendor-ID | LUID       |
|     |                                  |        |     |            | Device-ID | UUID       |
|     |                                  |        |     |            | Bus-ID    |            |
|---------------------------------------------------------------------------------------------|
| 0   | NVIDIA RTX A4000                 | Yes: 0 |     | 16622   MB | 10de      | 0          |
|     |                                  |        |     |            | 24b0      | 44ccd3c0.. |
|     |                                  |        |     |            | 6         |            |
|=============================================================================================|
| OS: 22.04.5 LTS (Jammy Jellyfish) ubuntu, Version: 22.04.5, Kernel: 6.8.0-48-generic
| Processor: AMD Ryzen 7 2700 Eight-Core Processor | Cores: 8 | Logical: 16
|---------------------------------------------------------------------------------------------|
| Total Memory (MB): 48078 | Free Memory: 40730
| Total Page/Swap (MB): 2047 | Free Page/Swap: 2046
|---------------------------------------------------------------------------------------------|
2024-11-17 03:38:43 [1,448ms] [Warning] [gpu.foundation.plugin] IOMMU is enabled.
2024-11-17 03:38:44 [2,239ms] [Warning] [carb.windowing-glfw.plugin] GLFW initialization failed.
2024-11-17 03:38:44 [2,239ms] [Warning] [carb] Failed to startup plugin carb.windowing-glfw.plugin (interfaces: [carb::windowing::IGLContext v1.0],[carb::windowing::IWindowing v1.4]) (impl: carb.windowing-glfw.plugin)
2024-11-17 03:38:44 [2,250ms] [Warning] [carb.windowing-glfw.plugin] GLFW initialization failed.
2024-11-17 03:38:44 [2,250ms] [Warning] [carb] Failed to startup plugin carb.windowing-glfw.plugin (interfaces: [carb::windowing::IGLContext v1.0],[carb::windowing::IWindowing v1.4]) (impl: carb.windowing-glfw.plugin)
2024-11-17 03:38:44 [2,259ms] [Warning] [carb.windowing-glfw.plugin] GLFW initialization failed.
2024-11-17 03:38:44 [2,259ms] [Warning] [carb] Failed to startup plugin carb.windowing-glfw.plugin (interfaces: [carb::windowing::IGLContext v1.0],[carb::windowing::IWindowing v1.4]) (impl: carb.windowing-glfw.plugin)
2024-11-17 03:38:44 [2,269ms] [Warning] [carb.windowing-glfw.plugin] GLFW initialization failed.
2024-11-17 03:38:44 [2,269ms] [Warning] [carb] Failed to startup plugin carb.windowing-glfw.plugin (interfaces: [carb::windowing::IGLContext v1.0],[carb::windowing::IWindowing v1.4]) (impl: carb.windowing-glfw.plugin)
2024-11-17 03:38:44 [2,271ms] [Warning] [carb.windowing-glfw.plugin] GLFW initialization failed.
2024-11-17 03:38:44 [2,271ms] [Warning] [carb] Failed to startup plugin carb.windowing-glfw.plugin (interfaces: [carb::windowing::IGLContext v1.0],[carb::windowing::IWindowing v1.4]) (impl: carb.windowing-glfw.plugin)
2024-11-17 03:38:44 [2,272ms] [Warning] [carb.windowing-glfw.plugin] GLFW initialization failed.
2024-11-17 03:38:44 [2,272ms] [Warning] [carb] Failed to startup plugin carb.windowing-glfw.plugin (interfaces: [carb::windowing::IGLContext v1.0],[carb::windowing::IWindowing v1.4]) (impl: carb.windowing-glfw.plugin)
2024-11-17 03:38:47 [5,563ms] [Warning] [omni.replicator.core.scripts.annotators] Annotator PostProcessDispatch is already registered, overwriting annotator template
formatted_tuple: (0.5, 0.5, 0.5, 0.5)
[INFO]: Parsing configuration from: omni.isaac.lab_tasks.direct.AT3R.AT3R_rolling_env:AT3RRoughEnvCfg
[INFO]: Parsing configuration from: /home/robot/IsaacLab_v1.2.0/IsaacLab/source/extensions/omni.isaac.lab_tasks/omni/isaac/lab_tasks/direct/AT3R/agents/rl_games_rough_ppo_cfg.yaml
[INFO] Logging experiment in directory: /home/robot/IsaacLab_v1.2.0/IsaacLab/logs/rl_games/AT3R_rough_direct
Setting seed: 42
[INFO]: Base environment:
    Environment device    : cuda:0
    Environment seed      : 42
    Physics step-size     : 0.005
    Rendering step-size   : 0.02
    Environment step-size : 0.02
[INFO] Generating terrains based on curriculum took : 1.709247 seconds
[INFO]: Time taken for scene creation : 5.009588 seconds
[INFO]: Scene manager:  <class InteractiveScene>
    Number of environments: 1024
    Environment spacing   : 4.0
    Source prim name      : /World/envs/env_0
    Global prim paths     : ['/World/ground']
    Replicate physics     : True
[INFO]: Starting the simulation. This may take a few seconds. Please wait...
2024-11-17 03:39:03 [21,512ms] [Warning] [omni.ujitso] UJITSO : Build storage validation failed for  Storage status: 1
2024-11-17 03:39:03 [21,512ms] [Error] [omni.physx.cooking.plugin] opRes == OperationResult::SUCCESS is false.
[INFO]: Time taken for simulation start : 58.231141 seconds
[INFO] Event Manager:  <EventManager> contains 1 active terms.
+--------------------------------------+
| Active Event Terms in Mode: 'startup' |
+----------+---------------------------+
|  Index   | Name                      |
+----------+---------------------------+
|    0     | physics_material          |
|    1     | add_base_mass             |
+----------+---------------------------+

[INFO]: Completed setting up the environment...
[INFO] Recording videos during training.
        video_folder: /home/robot/IsaacLab_v1.2.0/IsaacLab/logs/rl_games/AT3R_rough_direct/2024-11-17_12-38-52/videos/train
        step_trigger: lambda step: step % args_cli.video_interval == 0
        video_length: 400
        disable_logger: True
/home/robot/anaconda3/envs/iccar2025/lib/python3.10/site-packages/gymnasium/core.py:297: UserWarning: WARN: env.is_vector_env to get variables from other wrappers is deprecated and will be removed in v1.0, to get this variable you can do `env.unwrapped.is_vector_env` for environment variables or `env.get_attr('is_vector_env')` that will search the reminding wrappers.
  logger.warn(
self.seed = 42
Started to train
Exact experiment name requested from command line: 2024-11-17_12-38-52
seq_length: 4
current training device: cuda:0
/home/robot/anaconda3/envs/iccar2025/lib/python3.10/site-packages/rl_games/common/a2c_common.py:254: FutureWarning: `torch.cuda.amp.GradScaler(args...)` is deprecated. Please use `torch.amp.GradScaler('cuda', args...)` instead.
  self.scaler = torch.cuda.amp.GradScaler(enabled=self.mixed_precision)
build mlp: 252
RunningMeanStd:  (1,)
2024-11-17 03:40:00 [78,469ms] [Warning] [omni.syntheticdata.plugin] OgnSdPostRenderVarToHost : rendervar copy from texture directly to host buffer is counter-performant. Please use copy from texture to device buffer first.
/home/robot/anaconda3/envs/iccar2025/lib/python3.10/site-packages/rl_games/algos_torch/a2c_continuous.py:106: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  with torch.cuda.amp.autocast(enabled=self.mixed_precision):
fps step: 4462 fps step and policy inference: 4405 fps total: 4137 epoch: 1/5000 frames: 0
fps step: 6106 fps step and policy inference: 6017 fps total: 5972 epoch: 2/5000 frames: 24576
RandomOakForest commented 4 days ago

Thanks for posting this. We will review with the team.