Cartpole is slow in 2023.1.0

[sujitvasanth]

I just upgraded to Isaac Sim 2023.1.0 and therefore recloned the omnisaacgymenvs repo and reinstalled omniisaacgym through git as per the repo instructions. It does work but it has some strange/slow behaviour.

I get some omniverse gui windows transiently popping up and then omniverse hangs for 1-2 minutes at this point before running the rl example. I have taken a screenshot of the hanging:

the wanings in the screen shot are

Sim Device:  GPU
Task Device: cuda:0
RL device:  cuda:0
2023-10-29 09:00:04 [16,119ms] [Warning] [omni.client.python] Detected a blocking function. This will cause hitches or hangs in the UI. Please switch to the async version:
  File "/home/sujit/Desktop/OmniIsaacGymEnvs2023/omniisaacgymenvs/scripts/rlgames_train.py", line 150, in <module>
  File "/home/sujit/.local/share/ov/pkg/isaac_sim-2023.1.0/kit/python/lib/python3.10/site-packages/hydra/main.py", line 94, in decorated_main
  File "/home/sujit/.local/share/ov/pkg/isaac_sim-2023.1.0/kit/python/lib/python3.10/site-packages/hydra/_internal/utils.py", line 394, in _run_hydra
  File "/home/sujit/.local/share/ov/pkg/isaac_sim-2023.1.0/kit/python/lib/python3.10/site-packages/hydra/_internal/utils.py", line 457, in _run_app
  File "/home/sujit/.local/share/ov/pkg/isaac_sim-2023.1.0/kit/python/lib/python3.10/site-packages/hydra/_internal/utils.py", line 220, in run_and_report
  File "/home/sujit/.local/share/ov/pkg/isaac_sim-2023.1.0/kit/python/lib/python3.10/site-packages/hydra/_internal/utils.py", line 458, in <lambda>
  File "/home/sujit/.local/share/ov/pkg/isaac_sim-2023.1.0/kit/python/lib/python3.10/site-packages/hydra/_internal/hydra.py", line 119, in run
  File "/home/sujit/.local/share/ov/pkg/isaac_sim-2023.1.0/kit/python/lib/python3.10/site-packages/hydra/core/utils.py", line 186, in run_job
  File "/home/sujit/Desktop/OmniIsaacGymEnvs2023/omniisaacgymenvs/scripts/rlgames_train.py", line 121, in parse_hydra_configs
  File "/home/sujit/Desktop/OmniIsaacGymEnvs2023/omniisaacgymenvs/utils/task_util.py", line 103, in initialize_task
  File "/home/sujit/Desktop/OmniIsaacGymEnvs2023/omniisaacgymenvs/envs/vec_env_rlgames.py", line 47, in set_task
  File "/home/sujit/.local/share/ov/pkg/isaac_sim-2023.1.0/exts/omni.isaac.gym/omni/isaac/gym/vec_env/vec_env_base.py", line 126, in set_task
  File "/home/sujit/.local/share/ov/pkg/isaac_sim-2023.1.0/exts/omni.isaac.core/omni/isaac/core/world/world.py", line 276, in reset
  File "/home/sujit/Desktop/OmniIsaacGymEnvs2023/omniisaacgymenvs/tasks/cartpole.py", line 65, in set_up_scene
  File "/home/sujit/Desktop/OmniIsaacGymEnvs2023/omniisaacgymenvs/tasks/cartpole.py", line 83, in get_cartpole
  File "/home/sujit/Desktop/OmniIsaacGymEnvs2023/omniisaacgymenvs/robots/articulations/cartpole.py", line 54, in __init__
  File "/home/sujit/.local/share/ov/pkg/isaac_sim-2023.1.0/exts/omni.isaac.core/omni/isaac/core/utils/nucleus.py", line 504, in get_assets_root_path
  File "/home/sujit/.local/share/ov/pkg/isaac_sim-2023.1.0/exts/omni.isaac.core/omni/isaac/core/utils/nucleus.py", line 191, in check_server
  File "/home/sujit/.local/share/ov/pkg/isaac_sim-2023.1.0/kit/extscore/omni.client/omni/client/__init__.py", line 161, in stat

then after hanging for 1-2 minutes, i get this error and cartpole starts normally at the normal speed

2023-10-29 09:02:05 [137,263ms] [Warning] [omni.isaac.core.utils.nucleus] /persistent/app/omniverse/mountedDrives setting not found
2023-10-29 09:02:07 [139,036ms] [Warning] [omni.isaac.core.utils.nucleus] /persistent/app/omniverse/mountedDrives setting not found
~*~*~*
~*~*~* Direct GPU Helper:
~*~*~*   512 articulations, maxLinks=3
~*~*~*
self.seed = 42
Started to train
Exact experiment name requested from command line: Cartpole
Box(-1.0, 1.0, (1,), float32) Box(-inf, inf, (4,), float32)
seq_length: 4
current training device: cuda:0
build mlp: 4
RunningMeanStd:  (1,)
RunningMeanStd:  (4,)
[2023-10-29 09:02:09] Running RL reset
fps step: 30103 fps step and policy inference: 27797 fps total: 25915 epoch: 1/100 frames: 0
fps step: 35992 fps step and policy inference: 33289 fps total: 30690 epoch: 2/100 frames: 8192
fps step: 34550 fps step and policy inference: 31696 fps total: 29390 epoch: 3/100 frames: 16384
fps step: 34697 fps step and policy inference: 31565 fps total: 29357 epoch: 4/100 frames: 24576
fps step: 34479 fps step and policy inference: 31280 fps total: 29116 epoch: 5/100 frames: 32768
fps step: 34432 fps step and policy inference: 31383 fps total: 29321 epoch: 6/100 frames: 40960
fps step: 34310 fps step and policy inference: 31217 fps total: 29131 epoch: 7/100 frames: 49152
fps step: 34603 fps step and policy inference: 31570 fps total: 29363 epoch: 8/100 frames: 57344
fps step: 33794 fps step and policy inference: 30878 fps total: 28656 epoch: 9/100 frames: 65536
fps step: 33828 fps step and policy inference: 30696 fps total: 28656 epoch: 10/100 frames: 73728
fps step: 33990 fps step and policy inference: 31037 fps total: 28968 epoch: 11/100 frames: 81920
fps step: 34108 fps step and policy inference: 31240 fps total: 29119 epoch: 12/100 frames: 90112

my system is ubuntu 20.04, RTX 3090, 64 Gb RAM

How can I resolve the system hanging? also what is with the gui windows appearing for import and also the simulation window is not the same as 2021.2.1, why?

Opening files directly in Isaac Sim 2023.1.0 works fine and it has fixed issues I was having with usd robot file errors so I definitely need to upgrade

[kellyguo11]

Hi there, we are aware of the omni.client warning on startup, which is causing the startup process to become slower, and we are working on a fix for it. For now, you can try downloading the assets from the server and using the local paths to the assets instead of using get_assets_root_path(). The UI windows pop up as extensions are being loaded on startup. Once all extensions have finished loading and the scene has been created, only the Viewport window should be visible in the UI. Additional extensions can be loaded from the Windows > Extensions Manager window.

[sujitvasanth]

Thanks! I'm very grateful to have independent installations of 2022.2.1 and 2023.1.0 - its great to be able to work on the old version while experimenting on the new one - ~I have a couple of version of the repo to use in the different versions. I have got 2023.1.0 working at good speed now but would prefer not to waste resources on the server so please do fix this can we not just delete the line of code that does the sever check for single computer installations or give it a conditional flag in the task yaml?

[sujitvasanth]

I found a solution without requiring the installation of a local nucleus server You should have the omniverse cache installed

you can edit this file /home/*/.local/share/ov/pkg/isaac_sim-2023.1.0/exts/omni.isaac.core/omni/isaac/core/utils/nucleus.py change lines 178 to 198 which is the check server function to below

def check_server(server: str, path: str) -> bool:
    """Check a specific server for a path

    Args:
        server (str): Name of Nucleus server
        path (str): Path to search

    Returns:
        bool: True if folder is found
    """
    carb.log_info("Checking path: {}{}".format(server, path))
    # Increase hang detection timeout
    if "localhost" not in server:
        omni.client.set_hang_detection_time_ms(10000)
        result, _ = omni.client.stat("{}{}".format(server, path))
        if result == Result.OK:
            carb.log_info("Success: {}{}".format(server, path))
            return True
    carb.log_info("Failure: {}{} not accessible".format(server, path))
    return False

Isaac Sim 2023.1.0 now loads in 10 seconds on my system with no nucleus local server needed

For clarity the updated version is on the right

Isaac Sim 2023.1.0 now loads in 10 seconds on my system with no nucleus local server needed

[Maverobot]

@sujitvasanth 's solution works great! Thanks!

For isaac-sim-4.0.0, the file to edit has a different location:

~/.local/share/ov/pkg/isaac-sim-4.0.0/exts/omni.isaac.nucleus/omni/isaac/nucleus/nucleus.py