Open H-Hisamichi opened 1 week ago
Could you please confirm the version of Isaac Lab you are using? Have you tried this in the 1.2 release? Thanks!
Hello @RandomOakForest ,
I'm using Isaac Lab v1.2.0 and Isaac Sim v4.2.0. Thanks again!
Thank you for submitting this. We have take note of this error. If it is causing a crash on your side, please submit a bug report. Thanks.
Hello @RandomOakForest,
Thank you for your response. The current error code does not cause a crash. However, I am not sure if this error affects policy. I will provide more information about my environment. Isaac Sim: 4.2.0 Isaac Lab: 1.2.0 RL code: based on anymal_c(direct), modified for original hexapod.
The display of error codes has also been changed. The cause is unknown. Example of error code pattern 1:
[INFO] Using python from: /home/robot/anaconda3/envs/iccar2025/bin/python
[INFO][AppLauncher]: Loading experience file: /home/robot/IsaacLab_v1.2.0/IsaacLab/source/apps/isaaclab.python.headless.rendering.kit
Loading user config located at: '/home/robot/.local/share/ov/pkg/isaac-sim-4.2.0/kit/data/Kit/Isaac-Sim/4.2/user.config.json'
[Info] [carb] Logging to file: /home/robot/.local/share/ov/pkg/isaac-sim-4.2.0/kit/logs/Kit/Isaac-Sim/4.2/kit_20241117_101409.log
2024-11-17 01:14:10 [0ms] [Warning] [omni.kit.app.plugin] No crash reporter present, dumps uploading isn't available.
2024-11-17 01:14:10 [673ms] [Warning] [carb.windowing-glfw.plugin] GLFW initialization failed.
2024-11-17 01:14:10 [673ms] [Warning] [carb] Failed to startup plugin carb.windowing-glfw.plugin (interfaces: [carb::windowing::IGLContext v1.0],[carb::windowing::IWindowing v1.4]) (impl: carb.windowing-glfw.plugin)
2024-11-17 01:14:10 [701ms] [Warning] [omni.platforminfo.plugin] failed to open the default display. Can't verify X Server version.
|---------------------------------------------------------------------------------------------|
| Driver Version: 535.183.01 | Graphics API: Vulkan
|=============================================================================================|
| GPU | Name | Active | LDA | GPU Memory | Vendor-ID | LUID |
| | | | | | Device-ID | UUID |
| | | | | | Bus-ID | |
|---------------------------------------------------------------------------------------------|
| 0 | NVIDIA RTX A4000 | Yes: 0 | | 16622 MB | 10de | 0 |
| | | | | | 24b0 | 44ccd3c0.. |
| | | | | | 6 | |
|=============================================================================================|
| OS: 22.04.5 LTS (Jammy Jellyfish) ubuntu, Version: 22.04.5, Kernel: 6.8.0-48-generic
| Processor: AMD Ryzen 7 2700 Eight-Core Processor | Cores: 8 | Logical: 16
|---------------------------------------------------------------------------------------------|
| Total Memory (MB): 48078 | Free Memory: 41052
| Total Page/Swap (MB): 2047 | Free Page/Swap: 2047
|---------------------------------------------------------------------------------------------|
2024-11-17 01:14:11 [1,319ms] [Warning] [gpu.foundation.plugin] IOMMU is enabled.
2024-11-17 01:14:12 [2,183ms] [Warning] [carb.windowing-glfw.plugin] GLFW initialization failed.
2024-11-17 01:14:12 [2,183ms] [Warning] [carb] Failed to startup plugin carb.windowing-glfw.plugin (interfaces: [carb::windowing::IGLContext v1.0],[carb::windowing::IWindowing v1.4]) (impl: carb.windowing-glfw.plugin)
2024-11-17 01:14:12 [2,193ms] [Warning] [carb.windowing-glfw.plugin] GLFW initialization failed.
2024-11-17 01:14:12 [2,193ms] [Warning] [carb] Failed to startup plugin carb.windowing-glfw.plugin (interfaces: [carb::windowing::IGLContext v1.0],[carb::windowing::IWindowing v1.4]) (impl: carb.windowing-glfw.plugin)
2024-11-17 01:14:12 [2,201ms] [Warning] [carb.windowing-glfw.plugin] GLFW initialization failed.
2024-11-17 01:14:12 [2,201ms] [Warning] [carb] Failed to startup plugin carb.windowing-glfw.plugin (interfaces: [carb::windowing::IGLContext v1.0],[carb::windowing::IWindowing v1.4]) (impl: carb.windowing-glfw.plugin)
2024-11-17 01:14:12 [2,210ms] [Warning] [carb.windowing-glfw.plugin] GLFW initialization failed.
2024-11-17 01:14:12 [2,210ms] [Warning] [carb] Failed to startup plugin carb.windowing-glfw.plugin (interfaces: [carb::windowing::IGLContext v1.0],[carb::windowing::IWindowing v1.4]) (impl: carb.windowing-glfw.plugin)
2024-11-17 01:14:12 [2,212ms] [Warning] [carb.windowing-glfw.plugin] GLFW initialization failed.
2024-11-17 01:14:12 [2,212ms] [Warning] [carb] Failed to startup plugin carb.windowing-glfw.plugin (interfaces: [carb::windowing::IGLContext v1.0],[carb::windowing::IWindowing v1.4]) (impl: carb.windowing-glfw.plugin)
2024-11-17 01:14:12 [2,213ms] [Warning] [carb.windowing-glfw.plugin] GLFW initialization failed.
2024-11-17 01:14:12 [2,213ms] [Warning] [carb] Failed to startup plugin carb.windowing-glfw.plugin (interfaces: [carb::windowing::IGLContext v1.0],[carb::windowing::IWindowing v1.4]) (impl: carb.windowing-glfw.plugin)
2024-11-17 01:14:15 [5,363ms] [Warning] [omni.replicator.core.scripts.annotators] Annotator PostProcessDispatch is already registered, overwriting annotator template
formatted_tuple: (0.5, 0.5, 0.5, 0.5)
[INFO]: Parsing configuration from: omni.isaac.lab_tasks.direct.AT3R.AT3R_rolling_env:AT3RRoughEnvCfg
[INFO]: Parsing configuration from: /home/robot/IsaacLab_v1.2.0/IsaacLab/source/extensions/omni.isaac.lab_tasks/omni/isaac/lab_tasks/direct/AT3R/agents/rl_games_rough_ppo_cfg.yaml
[INFO] Logging experiment in directory: /home/robot/IsaacLab_v1.2.0/IsaacLab/logs/rl_games/AT3R_rough_direct
Setting seed: 42
[INFO]: Base environment:
Environment device : cuda:0
Environment seed : 42
Physics step-size : 0.005
Rendering step-size : 0.02
Environment step-size : 0.02
[INFO] Generating terrains based on curriculum took : 1.656922 seconds
[INFO]: Time taken for scene creation : 4.838669 seconds
[INFO]: Scene manager: <class InteractiveScene>
Number of environments: 1024
Environment spacing : 4.0
Source prim name : /World/envs/env_0
Global prim paths : ['/World/ground']
Replicate physics : True
[INFO]: Starting the simulation. This may take a few seconds. Please wait...
2024-11-17 01:14:30 [20,747ms] [Warning] [omni.ujitso] UJITSO : Build storage validation failed for Storage status: 1
2024-11-17 01:14:30 [20,747ms] [Error] [omni.physx.cooking.plugin] opRes == OperationResult::SUCCESS is false.
2024-11-17 01:14:30 [20,749ms] [Warning] [omni.ujitso] UJITSO : Build storage validation failed for Storage status: 1
2024-11-17 01:14:30 [20,749ms] [Error] [omni.physx.cooking.plugin] opRes == OperationResult::SUCCESS is false.
[INFO]: Time taken for simulation start : 57.431758 seconds
[INFO] Event Manager: <EventManager> contains 1 active terms.
+--------------------------------------+
| Active Event Terms in Mode: 'startup' |
+----------+---------------------------+
| Index | Name |
+----------+---------------------------+
| 0 | physics_material |
| 1 | add_base_mass |
+----------+---------------------------+
[INFO]: Completed setting up the environment...
[INFO] Recording videos during training.
video_folder: /home/robot/IsaacLab_v1.2.0/IsaacLab/logs/rl_games/AT3R_rough_direct/2024-11-17_10-14-20/videos/train
step_trigger: lambda step: step % args_cli.video_interval == 0
video_length: 400
disable_logger: True
/home/robot/anaconda3/envs/iccar2025/lib/python3.10/site-packages/gymnasium/core.py:297: UserWarning: WARN: env.is_vector_env to get variables from other wrappers is deprecated and will be removed in v1.0, to get this variable you can do `env.unwrapped.is_vector_env` for environment variables or `env.get_attr('is_vector_env')` that will search the reminding wrappers.
logger.warn(
self.seed = 42
Started to train
Exact experiment name requested from command line: 2024-11-17_10-14-20
seq_length: 4
current training device: cuda:0
/home/robot/anaconda3/envs/iccar2025/lib/python3.10/site-packages/rl_games/common/a2c_common.py:254: FutureWarning: `torch.cuda.amp.GradScaler(args...)` is deprecated. Please use `torch.amp.GradScaler('cuda', args...)` instead.
self.scaler = torch.cuda.amp.GradScaler(enabled=self.mixed_precision)
build mlp: 252
RunningMeanStd: (1,)
2024-11-17 01:15:27 [77,183ms] [Warning] [omni.syntheticdata.plugin] OgnSdPostRenderVarToHost : rendervar copy from texture directly to host buffer is counter-performant. Please use copy from texture to device buffer first.
/home/robot/anaconda3/envs/iccar2025/lib/python3.10/site-packages/rl_games/algos_torch/a2c_continuous.py:106: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
with torch.cuda.amp.autocast(enabled=self.mixed_precision):
fps step: 4422 fps step and policy inference: 4368 fps total: 4190 epoch: 1/5000 frames: 0
fps step: 6197 fps step and policy inference: 6106 fps total: 6055 epoch: 2/5000 frames: 24576
Example of error code pattern 2:
[INFO] Using python from: /home/robot/anaconda3/envs/iccar2025/bin/python
[INFO][AppLauncher]: Loading experience file: /home/robot/IsaacLab_v1.2.0/IsaacLab/source/apps/isaaclab.python.headless.rendering.kit
Loading user config located at: '/home/robot/.local/share/ov/pkg/isaac-sim-4.2.0/kit/data/Kit/Isaac-Sim/4.2/user.config.json'
[Info] [carb] Logging to file: /home/robot/.local/share/ov/pkg/isaac-sim-4.2.0/kit/logs/Kit/Isaac-Sim/4.2/kit_20241117_123841.log
2024-11-17 03:38:41 [0ms] [Warning] [omni.kit.app.plugin] No crash reporter present, dumps uploading isn't available.
2024-11-17 03:38:42 [744ms] [Warning] [carb.windowing-glfw.plugin] GLFW initialization failed.
2024-11-17 03:38:42 [744ms] [Warning] [carb] Failed to startup plugin carb.windowing-glfw.plugin (interfaces: [carb::windowing::IGLContext v1.0],[carb::windowing::IWindowing v1.4]) (impl: carb.windowing-glfw.plugin)
2024-11-17 03:38:42 [770ms] [Warning] [omni.platforminfo.plugin] failed to open the default display. Can't verify X Server version.
|---------------------------------------------------------------------------------------------|
| Driver Version: 535.183.01 | Graphics API: Vulkan
|=============================================================================================|
| GPU | Name | Active | LDA | GPU Memory | Vendor-ID | LUID |
| | | | | | Device-ID | UUID |
| | | | | | Bus-ID | |
|---------------------------------------------------------------------------------------------|
| 0 | NVIDIA RTX A4000 | Yes: 0 | | 16622 MB | 10de | 0 |
| | | | | | 24b0 | 44ccd3c0.. |
| | | | | | 6 | |
|=============================================================================================|
| OS: 22.04.5 LTS (Jammy Jellyfish) ubuntu, Version: 22.04.5, Kernel: 6.8.0-48-generic
| Processor: AMD Ryzen 7 2700 Eight-Core Processor | Cores: 8 | Logical: 16
|---------------------------------------------------------------------------------------------|
| Total Memory (MB): 48078 | Free Memory: 40730
| Total Page/Swap (MB): 2047 | Free Page/Swap: 2046
|---------------------------------------------------------------------------------------------|
2024-11-17 03:38:43 [1,448ms] [Warning] [gpu.foundation.plugin] IOMMU is enabled.
2024-11-17 03:38:44 [2,239ms] [Warning] [carb.windowing-glfw.plugin] GLFW initialization failed.
2024-11-17 03:38:44 [2,239ms] [Warning] [carb] Failed to startup plugin carb.windowing-glfw.plugin (interfaces: [carb::windowing::IGLContext v1.0],[carb::windowing::IWindowing v1.4]) (impl: carb.windowing-glfw.plugin)
2024-11-17 03:38:44 [2,250ms] [Warning] [carb.windowing-glfw.plugin] GLFW initialization failed.
2024-11-17 03:38:44 [2,250ms] [Warning] [carb] Failed to startup plugin carb.windowing-glfw.plugin (interfaces: [carb::windowing::IGLContext v1.0],[carb::windowing::IWindowing v1.4]) (impl: carb.windowing-glfw.plugin)
2024-11-17 03:38:44 [2,259ms] [Warning] [carb.windowing-glfw.plugin] GLFW initialization failed.
2024-11-17 03:38:44 [2,259ms] [Warning] [carb] Failed to startup plugin carb.windowing-glfw.plugin (interfaces: [carb::windowing::IGLContext v1.0],[carb::windowing::IWindowing v1.4]) (impl: carb.windowing-glfw.plugin)
2024-11-17 03:38:44 [2,269ms] [Warning] [carb.windowing-glfw.plugin] GLFW initialization failed.
2024-11-17 03:38:44 [2,269ms] [Warning] [carb] Failed to startup plugin carb.windowing-glfw.plugin (interfaces: [carb::windowing::IGLContext v1.0],[carb::windowing::IWindowing v1.4]) (impl: carb.windowing-glfw.plugin)
2024-11-17 03:38:44 [2,271ms] [Warning] [carb.windowing-glfw.plugin] GLFW initialization failed.
2024-11-17 03:38:44 [2,271ms] [Warning] [carb] Failed to startup plugin carb.windowing-glfw.plugin (interfaces: [carb::windowing::IGLContext v1.0],[carb::windowing::IWindowing v1.4]) (impl: carb.windowing-glfw.plugin)
2024-11-17 03:38:44 [2,272ms] [Warning] [carb.windowing-glfw.plugin] GLFW initialization failed.
2024-11-17 03:38:44 [2,272ms] [Warning] [carb] Failed to startup plugin carb.windowing-glfw.plugin (interfaces: [carb::windowing::IGLContext v1.0],[carb::windowing::IWindowing v1.4]) (impl: carb.windowing-glfw.plugin)
2024-11-17 03:38:47 [5,563ms] [Warning] [omni.replicator.core.scripts.annotators] Annotator PostProcessDispatch is already registered, overwriting annotator template
formatted_tuple: (0.5, 0.5, 0.5, 0.5)
[INFO]: Parsing configuration from: omni.isaac.lab_tasks.direct.AT3R.AT3R_rolling_env:AT3RRoughEnvCfg
[INFO]: Parsing configuration from: /home/robot/IsaacLab_v1.2.0/IsaacLab/source/extensions/omni.isaac.lab_tasks/omni/isaac/lab_tasks/direct/AT3R/agents/rl_games_rough_ppo_cfg.yaml
[INFO] Logging experiment in directory: /home/robot/IsaacLab_v1.2.0/IsaacLab/logs/rl_games/AT3R_rough_direct
Setting seed: 42
[INFO]: Base environment:
Environment device : cuda:0
Environment seed : 42
Physics step-size : 0.005
Rendering step-size : 0.02
Environment step-size : 0.02
[INFO] Generating terrains based on curriculum took : 1.709247 seconds
[INFO]: Time taken for scene creation : 5.009588 seconds
[INFO]: Scene manager: <class InteractiveScene>
Number of environments: 1024
Environment spacing : 4.0
Source prim name : /World/envs/env_0
Global prim paths : ['/World/ground']
Replicate physics : True
[INFO]: Starting the simulation. This may take a few seconds. Please wait...
2024-11-17 03:39:03 [21,512ms] [Warning] [omni.ujitso] UJITSO : Build storage validation failed for Storage status: 1
2024-11-17 03:39:03 [21,512ms] [Error] [omni.physx.cooking.plugin] opRes == OperationResult::SUCCESS is false.
[INFO]: Time taken for simulation start : 58.231141 seconds
[INFO] Event Manager: <EventManager> contains 1 active terms.
+--------------------------------------+
| Active Event Terms in Mode: 'startup' |
+----------+---------------------------+
| Index | Name |
+----------+---------------------------+
| 0 | physics_material |
| 1 | add_base_mass |
+----------+---------------------------+
[INFO]: Completed setting up the environment...
[INFO] Recording videos during training.
video_folder: /home/robot/IsaacLab_v1.2.0/IsaacLab/logs/rl_games/AT3R_rough_direct/2024-11-17_12-38-52/videos/train
step_trigger: lambda step: step % args_cli.video_interval == 0
video_length: 400
disable_logger: True
/home/robot/anaconda3/envs/iccar2025/lib/python3.10/site-packages/gymnasium/core.py:297: UserWarning: WARN: env.is_vector_env to get variables from other wrappers is deprecated and will be removed in v1.0, to get this variable you can do `env.unwrapped.is_vector_env` for environment variables or `env.get_attr('is_vector_env')` that will search the reminding wrappers.
logger.warn(
self.seed = 42
Started to train
Exact experiment name requested from command line: 2024-11-17_12-38-52
seq_length: 4
current training device: cuda:0
/home/robot/anaconda3/envs/iccar2025/lib/python3.10/site-packages/rl_games/common/a2c_common.py:254: FutureWarning: `torch.cuda.amp.GradScaler(args...)` is deprecated. Please use `torch.amp.GradScaler('cuda', args...)` instead.
self.scaler = torch.cuda.amp.GradScaler(enabled=self.mixed_precision)
build mlp: 252
RunningMeanStd: (1,)
2024-11-17 03:40:00 [78,469ms] [Warning] [omni.syntheticdata.plugin] OgnSdPostRenderVarToHost : rendervar copy from texture directly to host buffer is counter-performant. Please use copy from texture to device buffer first.
/home/robot/anaconda3/envs/iccar2025/lib/python3.10/site-packages/rl_games/algos_torch/a2c_continuous.py:106: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
with torch.cuda.amp.autocast(enabled=self.mixed_precision):
fps step: 4462 fps step and policy inference: 4405 fps total: 4137 epoch: 1/5000 frames: 0
fps step: 6106 fps step and policy inference: 6017 fps total: 5972 epoch: 2/5000 frames: 24576
Thanks for posting this. We will review with the team.
Hello everyone,
I encountered an unknown error while training my original Hexapod using reinforcement learning.
First, I get this warning:
[Warning] [omni.ujitso] UJITSO: Build storage validation failed for Storage status: 1
, and then the error[Error] [omni.physx.cooking.plugin] opRes == OperationResult::SUCCESS is false.
appears. The error occurred after the log[INFO]: Starting the simulation. This may take a few seconds. Please wait...
.Sometimes this error causes a buffer error, but other times the training continues without any issues. I couldn't find any information about this error. Does anyone know what this means?
Thank you!