search

isaac-sim / OmniIsaacGymEnvs

Reinforcement Learning Environments for Omniverse Isaac Gym
Other
764 stars 202 forks source link

The Omniverse Isaac Gym is painfully slow. #103

Closed Guanbin-Huang closed 7 months ago

Guanbin-Huang commented 7 months ago

The Omniverse isaac gym is very slow. It takes a long time to run a training session for the following:

I have tried two commands, but both of them take a significant amount of time to execute.

PYTHON_PATH scripts/rlgames_train.py task=Humanoid I managed to execute it successfully once. However, I encountered an error at that time, but I didn’t log the details of the error. So I decided to log the output into a log.txt file.

PYTHON_PATH scripts/rlgames_train.py task=Humanoid > log.txt 2>&1 (takes even longer to execute, I didn't wait until I see the result so I'm not sure is working or not in this case) Nevertheless, it seems to be stuck at this point. image image image The most important CPU information for understanding the performance of your CPU are:

Architecture: x86_64 - CPU op-mode(s): 32-bit, 64-bit - Number of CPU(s): 32 - Threads per core: 2 - Cores per socket: 16 - Model name: AMD Ryzen 9 5950X 16-Core Processor - CPU frequency boost: enabled - CPU MHz (current frequency): 2200.000 - CPU max MHz: 3400.0000 - CPU min MHz: 2200.0000 - L1d cache: 512 KiB - L1i cache: 512 KiB

Guanbin-Huang commented 7 months ago

Here's one more image for supplementation

image
kellyguo11 commented 7 months ago

Hi there, if you are running training runs with the full 4096 environments, it is recommended to run in headless mode with PYTHON_PATH scripts/rlgames_train.py task=Humanoid headless=True. This will reduce both startup and simulation time significantly. Then, you can visualize the results of the training with inference runs using test=True checkpoint=/path/to/checkpoint and a smaller number of environments num_envs=10. If you are not already on the latest isaac sim release, please also try with the latest Isaac Sim 2023.1.0-hotfix.1 release.

Guanbin-Huang commented 7 months ago

Hi there, if you are running training runs with the full 4096 environments, it is recommended to run in headless mode with PYTHON_PATH scripts/rlgames_train.py task=Humanoid headless=True. This will reduce both startup and simulation time significantly. Then, you can visualize the results of the training with inference runs using test=True checkpoint=/path/to/checkpoint and a smaller number of environments num_envs=10. If you are not already on the latest isaac sim release, please also try with the latest Isaac Sim 2023.1.0-hotfix.1 release.

Great, I'll try that!