search

isaac-sim / OmniIsaacGymEnvs

Reinforcement Learning Environments for Omniverse Isaac Gym
Other
848 stars 218 forks source link

Segmentation fault in Docker -> Cartpole assertion error #120

Closed Cdfghglz closed 10 months ago

Cdfghglz commented 10 months ago

Hi, I am unable to run the isaac-sim inside Docker

cdf@xps15:~/ws/pr_omni_isaac_gym_envs$  (main)./docker/run_docker.sh 
docker: Error response from daemon: Conflict. The container name "/isaac-sim-oige" is already in use by container "46783acb7a7b1cfbe70fc53989462c3ac653a0be1aea47989ed16e1da589b841". You have to remove (or rename) that container to be able to reuse that name.
See 'docker run --help'.
Obtaining file:///workspace/omniisaacgymenvs
Requirement already satisfied: numpy==1.23.5 in /isaac-sim/kit/exts/omni.kit.pip_archive/pip_prebundle (from omniisaacgymenvs==2023.1.0b0) (1.23.5)
Requirement already satisfied: protobuf==3.20.2 in /isaac-sim/kit/python/lib/python3.10/site-packages (from omniisaacgymenvs==2023.1.0b0) (3.20.2)
Requirement already satisfied: omegaconf==2.3.0 in /isaac-sim/kit/python/lib/python3.10/site-packages (from omniisaacgymenvs==2023.1.0b0) (2.3.0)
Requirement already satisfied: hydra-core==1.3.2 in /isaac-sim/kit/python/lib/python3.10/site-packages (from omniisaacgymenvs==2023.1.0b0) (1.3.2)
Requirement already satisfied: urllib3==1.26.16 in /isaac-sim/kit/exts/omni.kit.pip_archive/pip_prebundle (from omniisaacgymenvs==2023.1.0b0) (1.26.16)
Requirement already satisfied: rl-games==1.6.1 in /isaac-sim/kit/python/lib/python3.10/site-packages (from omniisaacgymenvs==2023.1.0b0) (1.6.1)
Requirement already satisfied: packaging in /isaac-sim/exts/omni.isaac.core_archive/pip_prebundle (from hydra-core==1.3.2->omniisaacgymenvs==2023.1.0b0) (23.0)
Requirement already satisfied: antlr4-python3-runtime==4.9.* in /isaac-sim/kit/python/lib/python3.10/site-packages (from hydra-core==1.3.2->omniisaacgymenvs==2023.1.0b0) (4.9.3)
Requirement already satisfied: PyYAML>=5.1.0 in /isaac-sim/exts/omni.pip.compute/pip_prebundle (from omegaconf==2.3.0->omniisaacgymenvs==2023.1.0b0) (6.0.1)
Requirement already satisfied: tensorboard<3.0.0,>=2.8.0 in /isaac-sim/kit/python/lib/python3.10/site-packages (from rl-games==1.6.1->omniisaacgymenvs==2023.1.0b0) (2.15.1)
Requirement already satisfied: opencv-python<5.0.0,>=4.5.5 in /isaac-sim/kit/python/lib/python3.10/site-packages (from rl-games==1.6.1->omniisaacgymenvs==2023.1.0b0) (4.8.1.78)
Requirement already satisfied: tensorboardX<3.0,>=2.5 in /isaac-sim/kit/python/lib/python3.10/site-packages (from rl-games==1.6.1->omniisaacgymenvs==2023.1.0b0) (2.6.2.2)
Requirement already satisfied: wandb<0.13.0,>=0.12.11 in /isaac-sim/kit/python/lib/python3.10/site-packages (from rl-games==1.6.1->omniisaacgymenvs==2023.1.0b0) (0.12.21)
Requirement already satisfied: psutil<6.0.0,>=5.9.0 in /isaac-sim/kit/python/lib/python3.10/site-packages (from rl-games==1.6.1->omniisaacgymenvs==2023.1.0b0) (5.9.6)
Requirement already satisfied: setproctitle<2.0.0,>=1.2.2 in /isaac-sim/kit/python/lib/python3.10/site-packages (from rl-games==1.6.1->omniisaacgymenvs==2023.1.0b0) (1.3.3)
Requirement already satisfied: gym[classic-control]<0.24.0,>=0.23.0 in /isaac-sim/kit/python/lib/python3.10/site-packages (from rl-games==1.6.1->omniisaacgymenvs==2023.1.0b0) (0.23.1)
WARNING: gym 0.23.1 does not provide the extra 'classic-control'
Requirement already satisfied: cloudpickle>=1.2.0 in /isaac-sim/kit/python/lib/python3.10/site-packages (from gym[classic-control]<0.24.0,>=0.23.0->rl-games==1.6.1->omniisaacgymenvs==2023.1.0b0) (3.0.0)
Requirement already satisfied: gym-notices>=0.0.4 in /isaac-sim/kit/python/lib/python3.10/site-packages (from gym[classic-control]<0.24.0,>=0.23.0->rl-games==1.6.1->omniisaacgymenvs==2023.1.0b0) (0.0.8)
Requirement already satisfied: tensorboard-data-server<0.8.0,>=0.7.0 in /isaac-sim/kit/python/lib/python3.10/site-packages (from tensorboard<3.0.0,>=2.8.0->rl-games==1.6.1->omniisaacgymenvs==2023.1.0b0) (0.7.2)
Requirement already satisfied: grpcio>=1.48.2 in /isaac-sim/kit/python/lib/python3.10/site-packages (from tensorboard<3.0.0,>=2.8.0->rl-games==1.6.1->omniisaacgymenvs==2023.1.0b0) (1.60.0)
Requirement already satisfied: google-auth<3,>=1.6.3 in /isaac-sim/kit/python/lib/python3.10/site-packages (from tensorboard<3.0.0,>=2.8.0->rl-games==1.6.1->omniisaacgymenvs==2023.1.0b0) (2.25.2)
Requirement already satisfied: requests<3,>=2.21.0 in /isaac-sim/exts/omni.pip.cloud/pip_prebundle (from tensorboard<3.0.0,>=2.8.0->rl-games==1.6.1->omniisaacgymenvs==2023.1.0b0) (2.31.0)
Requirement already satisfied: werkzeug>=1.0.1 in /isaac-sim/kit/python/lib/python3.10/site-packages (from tensorboard<3.0.0,>=2.8.0->rl-games==1.6.1->omniisaacgymenvs==2023.1.0b0) (3.0.1)
Requirement already satisfied: google-auth-oauthlib<2,>=0.5 in /isaac-sim/kit/python/lib/python3.10/site-packages (from tensorboard<3.0.0,>=2.8.0->rl-games==1.6.1->omniisaacgymenvs==2023.1.0b0) (1.2.0)
Requirement already satisfied: setuptools>=41.0.0 in /isaac-sim/kit/exts/omni.kit.pip_archive/pip_prebundle (from tensorboard<3.0.0,>=2.8.0->rl-games==1.6.1->omniisaacgymenvs==2023.1.0b0) (68.0.0)
Requirement already satisfied: markdown>=2.6.8 in /isaac-sim/kit/python/lib/python3.10/site-packages (from tensorboard<3.0.0,>=2.8.0->rl-games==1.6.1->omniisaacgymenvs==2023.1.0b0) (3.5.1)
Requirement already satisfied: absl-py>=0.4 in /isaac-sim/kit/python/lib/python3.10/site-packages (from tensorboard<3.0.0,>=2.8.0->rl-games==1.6.1->omniisaacgymenvs==2023.1.0b0) (2.0.0)
Requirement already satisfied: six>1.9 in /isaac-sim/kit/exts/omni.kit.pip_archive/pip_prebundle (from tensorboard<3.0.0,>=2.8.0->rl-games==1.6.1->omniisaacgymenvs==2023.1.0b0) (1.16.0)
Requirement already satisfied: pyasn1-modules>=0.2.1 in /isaac-sim/kit/python/lib/python3.10/site-packages (from google-auth<3,>=1.6.3->tensorboard<3.0.0,>=2.8.0->rl-games==1.6.1->omniisaacgymenvs==2023.1.0b0) (0.3.0)
Requirement already satisfied: cachetools<6.0,>=2.0.0 in /isaac-sim/kit/python/lib/python3.10/site-packages (from google-auth<3,>=1.6.3->tensorboard<3.0.0,>=2.8.0->rl-games==1.6.1->omniisaacgymenvs==2023.1.0b0) (5.3.2)
Requirement already satisfied: rsa<5,>=3.1.4 in /isaac-sim/kit/python/lib/python3.10/site-packages (from google-auth<3,>=1.6.3->tensorboard<3.0.0,>=2.8.0->rl-games==1.6.1->omniisaacgymenvs==2023.1.0b0) (4.9)
Requirement already satisfied: requests-oauthlib>=0.7.0 in /isaac-sim/exts/omni.pip.cloud/pip_prebundle (from google-auth-oauthlib<2,>=0.5->tensorboard<3.0.0,>=2.8.0->rl-games==1.6.1->omniisaacgymenvs==2023.1.0b0) (1.3.1)
Requirement already satisfied: pyasn1<0.6.0,>=0.4.6 in /isaac-sim/kit/python/lib/python3.10/site-packages (from pyasn1-modules>=0.2.1->google-auth<3,>=1.6.3->tensorboard<3.0.0,>=2.8.0->rl-games==1.6.1->omniisaacgymenvs==2023.1.0b0) (0.5.1)
Requirement already satisfied: charset-normalizer<4,>=2 in /isaac-sim/kit/exts/omni.kit.pip_archive/pip_prebundle (from requests<3,>=2.21.0->tensorboard<3.0.0,>=2.8.0->rl-games==1.6.1->omniisaacgymenvs==2023.1.0b0) (2.1.1)
Requirement already satisfied: idna<4,>=2.5 in /isaac-sim/kit/exts/omni.kit.pip_archive/pip_prebundle (from requests<3,>=2.21.0->tensorboard<3.0.0,>=2.8.0->rl-games==1.6.1->omniisaacgymenvs==2023.1.0b0) (3.4)
Requirement already satisfied: certifi>=2017.4.17 in /isaac-sim/kit/exts/omni.kit.pip_archive/pip_prebundle (from requests<3,>=2.21.0->tensorboard<3.0.0,>=2.8.0->rl-games==1.6.1->omniisaacgymenvs==2023.1.0b0) (2023.5.7)
Requirement already satisfied: oauthlib>=3.0.0 in /isaac-sim/exts/omni.pip.cloud/pip_prebundle (from requests-oauthlib>=0.7.0->google-auth-oauthlib<2,>=0.5->tensorboard<3.0.0,>=2.8.0->rl-games==1.6.1->omniisaacgymenvs==2023.1.0b0) (3.2.2)
Requirement already satisfied: promise<3,>=2.0 in /isaac-sim/kit/python/lib/python3.10/site-packages (from wandb<0.13.0,>=0.12.11->rl-games==1.6.1->omniisaacgymenvs==2023.1.0b0) (2.3)
Requirement already satisfied: GitPython>=1.0.0 in /isaac-sim/kit/python/lib/python3.10/site-packages (from wandb<0.13.0,>=0.12.11->rl-games==1.6.1->omniisaacgymenvs==2023.1.0b0) (3.1.40)
Requirement already satisfied: docker-pycreds>=0.4.0 in /isaac-sim/kit/python/lib/python3.10/site-packages (from wandb<0.13.0,>=0.12.11->rl-games==1.6.1->omniisaacgymenvs==2023.1.0b0) (0.4.0)
Requirement already satisfied: pathtools in /isaac-sim/kit/exts/omni.kit.pip_archive/pip_prebundle (from wandb<0.13.0,>=0.12.11->rl-games==1.6.1->omniisaacgymenvs==2023.1.0b0) (0.1.2)
Requirement already satisfied: shortuuid>=0.5.0 in /isaac-sim/kit/python/lib/python3.10/site-packages (from wandb<0.13.0,>=0.12.11->rl-games==1.6.1->omniisaacgymenvs==2023.1.0b0) (1.0.11)
Requirement already satisfied: sentry-sdk>=1.0.0 in /isaac-sim/kit/exts/omni.kit.pip_archive/pip_prebundle (from wandb<0.13.0,>=0.12.11->rl-games==1.6.1->omniisaacgymenvs==2023.1.0b0) (1.14.0)
Requirement already satisfied: Click!=8.0.0,>=7.0 in /isaac-sim/kit/exts/omni.kit.pip_archive/pip_prebundle (from wandb<0.13.0,>=0.12.11->rl-games==1.6.1->omniisaacgymenvs==2023.1.0b0) (8.1.3)
Requirement already satisfied: gitdb<5,>=4.0.1 in /isaac-sim/kit/python/lib/python3.10/site-packages (from GitPython>=1.0.0->wandb<0.13.0,>=0.12.11->rl-games==1.6.1->omniisaacgymenvs==2023.1.0b0) (4.0.11)
Requirement already satisfied: smmap<6,>=3.0.1 in /isaac-sim/kit/python/lib/python3.10/site-packages (from gitdb<5,>=4.0.1->GitPython>=1.0.0->wandb<0.13.0,>=0.12.11->rl-games==1.6.1->omniisaacgymenvs==2023.1.0b0) (5.0.1)
Requirement already satisfied: MarkupSafe>=2.1.1 in /isaac-sim/exts/omni.isaac.core_archive/pip_prebundle (from werkzeug>=1.0.1->tensorboard<3.0.0,>=2.8.0->rl-games==1.6.1->omniisaacgymenvs==2023.1.0b0) (2.1.3)
Installing collected packages: omniisaacgymenvs
  Attempting uninstall: omniisaacgymenvs
    Found existing installation: omniisaacgymenvs 2023.1.0b0
    Uninstalling omniisaacgymenvs-2023.1.0b0:
      Successfully uninstalled omniisaacgymenvs-2023.1.0b0
  Running setup.py develop for omniisaacgymenvs
Successfully installed omniisaacgymenvs
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv                                                                                           
WARNING: You are using pip version 21.2.1+nv1; however, version 23.3.1 is available.
You should consider upgrading via the '/isaac-sim/kit/python/bin/python3 -m pip install --upgrade pip' command.                                                                                                 
root@xps15:/workspace/omniisaacgymenvs/omniisaacgymenvs# /isaac-sim/python.sh cripts/rlgames_train.py task=Cartpole
/isaac-sim/kit/python/bin/python3: can't open file '/workspace/omniisaacgymenvs/omniisaacgymenvs/cripts/rlgames_train.py': [Errno 2] No such file or directory
There was an error running python
root@xps15:/workspace/omniisaacgymenvs/omniisaacgymenvs# /isaac-sim/python.sh scripts/rlgames_train.py task=Cartpole
/isaac-sim/extscache/omni.pip.torch-2_0_1-2.0.2+105.1.lx64/torch-2-0-1/torch/utils/tensorboard/__init__.py:4: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
  if not hasattr(tensorboard, "__version__") or LooseVersion(
Starting kit application with the following args:  ['/isaac-sim/exts/omni.isaac.kit/omni/isaac/kit/simulation_app.py', '/isaac-sim/apps/omni.isaac.sim.python.gym.kit', '--/app/tokens/exe-path=/isaac-sim/kit', '--/persistent/app/viewport/displayOptions=3094', '--/rtx/materialDb/syncLoads=True', '--/rtx/hydra/materialSyncLoads=True', '--/omni.kit.plugin/syncUsdLoads=True', '--/app/renderer/resolution/width=1280', '--/app/renderer/resolution/height=720', '--/app/window/width=1440', '--/app/window/height=900', '--/renderer/multiGpu/enabled=True', '--/app/fastShutdown=True', '--ext-folder', '/isaac-sim/exts', '--ext-folder', '/isaac-sim/apps', '--/physics/cudaDevice=0', '--portable', '--allow-root']
Passing the following args to the base kit application:  ['task=Cartpole']
[Info] [carb] Logging to file: /isaac-sim/kit/logs/Kit/omni.isaac.sim.python.gym/2023.1/kit_20231215_111135.log
2023-12-15 11:11:35 [0ms] [Warning] [omni.kit.app.plugin] No crash reporter present, dumps uploading isn't available.
2023-12-15 11:11:35 [18ms] [Warning] [omni.ext.plugin] [ext: omni.kit.converter.cad_core-200.0.0-rc.3+105.0.lx64.r.cp310] Built using kit version: 105.0. Current version: 105.1. It is considered compatible, but building with a newer version is recommended.
2023-12-15 11:11:35 [18ms] [Warning] [omni.ext.plugin] [ext: omni.kit.converter.cad-200.0.0-rc.4+105.0] Built using kit version: 105.0. Current version: 105.1. It is considered compatible, but building with a newer version is recommended.
2023-12-15 11:11:35 [19ms] [Warning] [omni.ext.plugin] [ext: omni.kit.sequencer.core-103.4.1+105.0] Built using kit version: 105.0. Current version: 105.1. It is considered compatible, but building with a newer version is recommended.
2023-12-15 11:11:35 [20ms] [Warning] [omni.ext.plugin] [ext: omni.kit.sequencer.usd-103.4.2+105.0] Built using kit version: 105.0. Current version: 105.1. It is considered compatible, but building with a newer version is recommended.
2023-12-15 11:11:35 [21ms] [Warning] [omni.ext.plugin] [ext: omni.kit.widget.timeline-105.0.1+105.0] Built using kit version: 105.0. Current version: 105.1. It is considered compatible, but building with a newer version is recommended.
2023-12-15 11:11:35 [22ms] [Warning] [omni.ext.plugin] [ext: omni.kit.window.sequencer-103.4.2-dev.3+105.0] Built using kit version: 105.0. Current version: 105.1. It is considered compatible, but building with a newer version is recommended.
2023-12-15 11:11:35 [23ms] [Warning] [omni.ext.plugin] [ext: omni.paint.brush.attributes-1.3.1+105.0] Built using kit version: 105.0. Current version: 105.1. It is considered compatible, but building with a newer version is recommended.
2023-12-15 11:11:35 [25ms] [Warning] [omni.ext.plugin] [ext: omni.usd.schema.sequence-2.3.0+105.0.lx64.r.cp310] Built using kit version: 105.0. Current version: 105.1. It is considered compatible, but building with a newer version is recommended.
[0.074s] [ext: omni.kit.async_engine-0.0.0] startup
[1.180s] [ext: omni.assets.plugins-0.0.0] startup
[1.188s] [ext: omni.stats-0.0.0] startup
[1.191s] [ext: omni.client-1.0.1] startup
[1.224s] [ext: omni.gpu_foundation-0.0.0] startup
[1.271s] [ext: omni.rtx.shadercache.vulkan-1.0.0] startup
[1.276s] [ext: carb.windowing.plugins-1.0.0] startup
2023-12-15 11:11:37 [1,253ms] [Warning] [carb.windowing-glfw.plugin] GLFW initialization failed.
2023-12-15 11:11:37 [1,253ms] [Warning] [carb] Failed to startup plugin carb.windowing-glfw.plugin (interfaces: [carb::windowing::IGLContext v1.0],[carb::windowing::IWindowing v1.4]) (impl: carb.windowing-glfw.plugin)
[1.280s] [ext: omni.kit.renderer.init-0.0.0] startup
2023-12-15 11:11:37 [1,283ms] [Warning] [omni.platforminfo.plugin] failed to open the default display.  Can't verify X Server version.
2023-12-15 11:11:38 [2,948ms] [Warning] [carb.windowing-glfw.plugin] GLFW initialization failed.
2023-12-15 11:11:38 [2,948ms] [Warning] [carb] Failed to startup plugin carb.windowing-glfw.plugin (interfaces: [carb::windowing::IGLContext v1.0],[carb::windowing::IWindowing v1.4]) (impl: carb.windowing-glfw.plugin)

|---------------------------------------------------------------------------------------------|
| Driver Version: 535.129.03    | Graphics API: Vulkan
|=============================================================================================|
| GPU | Name                             | Active | LDA | GPU Memory | Vendor-ID | LUID       |
|     |                                  |        |     |            | Device-ID | UUID       |
|     |                                  |        |     |            | Bus-ID    |            |
|---------------------------------------------------------------------------------------------|
| 0   | NVIDIA GeForce RTX 4060 Laptop.. | Yes: 0 |     | 8188    MB | 10de      | 0          |
|     |                                  |        |     |            | 28a0      | 09391f11.. |
|     |                                  |        |     |            | 1         |            |
|=============================================================================================|
| OS: 22.04.3 LTS (Jammy Jellyfish) ubuntu, Version: 22.04.3, Kernel: 6.0.12-76060012-generic
| Processor: 13th Gen Intel(R) Core(TM) i7-13700H | Cores: 14 | Logical: 28
|---------------------------------------------------------------------------------------------|
| Total Memory (MB): 15682 | Free Memory: 10417
| Total Page/Swap (MB): 39746 | Free Page/Swap: 34583
|---------------------------------------------------------------------------------------------|
[3.583s] [ext: omni.kit.pipapi-0.0.0] startup
[3.585s] [ext: omni.kit.pip_archive-0.0.0] startup
[3.585s] [ext: omni.pip.compute-1.2.0] startup
[3.586s] [ext: omni.pip.torch-2_0_1-2.0.2] startup
[3.629s] [ext: omni.pip.cloud-1.0.1] startup
[3.631s] [ext: omni.isaac.core_archive-2.2.1] startup
[3.631s] [ext: omni.kit.telemetry-0.5.0] startup
[3.664s] [ext: omni.isaac.ml_archive-1.1.3] startup
[3.666s] [ext: omni.mtlx-0.1.0] startup
[3.668s] [ext: omni.usd.config-1.0.3] startup
[3.685s] [ext: omni.gpucompute.plugins-0.0.0] startup
[3.688s] [ext: omni.usd.libs-1.0.0] startup
[3.958s] [ext: omni.kit.loop-isaac-1.1.0] startup
[3.960s] [ext: omni.kit.test-0.0.0] startup
[3.962s] [ext: omni.usd.schema.omnigraph-1.0.0] startup
[4.185s] [ext: omni.usd.schema.physics-0.0.0] startup
[4.239s] [ext: omni.usd.schema.audio-0.0.0] startup
[4.245s] [ext: omni.usd.schema.physx-0.0.0] startup
[4.296s] [ext: omni.usd.schema.semantics-0.0.0] startup
[4.312s] [ext: omni.usd.schema.anim-0.0.0] startup
[4.345s] [ext: omni.usd.schema.omniscripting-1.0.0] startup
[4.355s] [ext: omni.usd.schema.geospatial-0.0.0] startup
[4.360s] [ext: omni.appwindow-1.1.5] startup
2023-12-15 11:11:40 [4,336ms] [Warning] [carb.windowing-glfw.plugin] GLFW initialization failed.
2023-12-15 11:11:40 [4,336ms] [Warning] [carb] Failed to startup plugin carb.windowing-glfw.plugin (interfaces: [carb::windowing::IGLContext v1.0],[carb::windowing::IWindowing v1.4]) (impl: carb.windowing-glfw.plugin)
2023-12-15 11:11:40 [4,337ms] [Warning] [carb.windowing-glfw.plugin] GLFW initialization failed.
2023-12-15 11:11:40 [4,337ms] [Warning] [carb] Failed to startup plugin carb.windowing-glfw.plugin (interfaces: [carb::windowing::IGLContext v1.0],[carb::windowing::IWindowing v1.4]) (impl: carb.windowing-glfw.plugin)
2023-12-15 11:11:40 [4,337ms] [Error] [omni.appwindow.plugin] Failed to acquire IWindowing interface
[4.366s] [ext: omni.kit.renderer.core-0.0.0] startup
Fatal Python error: Segmentation fault

Current thread 0x00007f8e7f90eb80 (most recent call first):
  File "/isaac-sim/exts/omni.isaac.kit/omni/isaac/kit/simulation_app.py", line 303 in _start_app
  File "/isaac-sim/exts/omni.isaac.kit/omni/isaac/kit/simulation_app.py", line 192 in __init__
  File "/isaac-sim/exts/omni.isaac.gym/omni/isaac/gym/vec_env/vec_env_base.py", line 56 in __init__
  File "/workspace/omniisaacgymenvs/omniisaacgymenvs/scripts/rlgames_train.py", line 98 in parse_hydra_configs
  File "/isaac-sim/kit/python/lib/python3.10/site-packages/hydra/core/utils.py", line 186 in run_job
  File "/isaac-sim/kit/python/lib/python3.10/site-packages/hydra/_internal/hydra.py", line 119 in run
  File "/isaac-sim/kit/python/lib/python3.10/site-packages/hydra/_internal/utils.py", line 458 in <lambda>
  File "/isaac-sim/kit/python/lib/python3.10/site-packages/hydra/_internal/utils.py", line 220 in run_and_report
  File "/isaac-sim/kit/python/lib/python3.10/site-packages/hydra/_internal/utils.py", line 457 in _run_app
  File "/isaac-sim/kit/python/lib/python3.10/site-packages/hydra/_internal/utils.py", line 394 in _run_hydra
  File "/isaac-sim/kit/python/lib/python3.10/site-packages/hydra/main.py", line 94 in decorated_main
  File "/workspace/omniisaacgymenvs/omniisaacgymenvs/scripts/rlgames_train.py", line 150 in <module>

Extension modules: yaml._yaml, numpy.core._multiarray_umath, numpy.core._multiarray_tests, numpy.linalg._umath_linalg, numpy.fft._pocketfft_internal, numpy.random._common, numpy.random.bit_generator, numpy.random._bounded_integers, numpy.random._mt19937, numpy.random.mtrand, numpy.random._philox, numpy.random._pcg64, numpy.random._sfc64, numpy.random._generator, torch._C, torch._C._fft, torch._C._linalg, torch._C._nested, torch._C._nn, torch._C._sparse, torch._C._special, google.protobuf.pyext._message, psutil._psutil_linux, psutil._psutil_posix (total: 24)
/isaac-sim/python.sh: line 41:   198 Segmentation fault      (core dumped) $python_exe "$@" $args
There was an error running python

Any help is appreciated

Toni-SM commented 10 months ago

Hi @Cdfghglz

Since you are working in a docker container, try passing the argument headless=True to the training script

Cdfghglz commented 10 months ago

Thank you for the reply, I got further: 

root@xps15:/workspace/omniisaacgymenvs/omniisaacgymenvs# /isaac-sim/python.sh scripts/rlgames_train.py task=Cartpole headless=True
/isaac-sim/extscache/omni.pip.torch-2_0_1-2.0.2+105.1.lx64/torch-2-0-1/torch/utils/tensorboard/__init__.py:4: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
  if not hasattr(tensorboard, "__version__") or LooseVersion(
Starting kit application with the following args:  ['/isaac-sim/exts/omni.isaac.kit/omni/isaac/kit/simulation_app.py', '/isaac-sim/apps/omni.isaac.sim.python.gym.headless.kit', '--/app/tokens/exe-path=/isaac-sim/kit', '--/persistent/app/viewport/displayOptions=3094', '--/rtx/materialDb/syncLoads=True', '--/rtx/hydra/materialSyncLoads=True', '--/omni.kit.plugin/syncUsdLoads=True', '--/app/renderer/resolution/width=1280', '--/app/renderer/resolution/height=720', '--/app/window/width=1440', '--/app/window/height=900', '--/renderer/multiGpu/enabled=True', '--/app/fastShutdown=True', '--ext-folder', '/isaac-sim/exts', '--ext-folder', '/isaac-sim/apps', '--/physics/cudaDevice=0', '--portable', '--no-window', '--allow-root']
Passing the following args to the base kit application:  ['task=Cartpole', 'headless=True']
Loading user config located at: '/isaac-sim/kit/data/Kit/Isaac-Sim/2023.1/user.config.json'
[Info] [carb] Logging to file: /isaac-sim/kit/logs/Kit/Isaac-Sim/2023.1/kit_20231219_075628.log
2023-12-19 07:56:28 [0ms] [Warning] [omni.kit.app.plugin] No crash reporter present, dumps uploading isn't available.
2023-12-19 07:56:28 [17ms] [Warning] [omni.ext.plugin] [ext: omni.kit.converter.cad-200.0.0-rc.4+105.0] Built using kit version: 105.0. Current version: 105.1. It is considered compatible, but building with a newer version is recommended.
2023-12-19 07:56:28 [17ms] [Warning] [omni.ext.plugin] [ext: omni.kit.converter.cad_core-200.0.0-rc.3+105.0.lx64.r.cp310] Built using kit version: 105.0. Current version: 105.1. It is considered compatible, but building with a newer version is recommended.
2023-12-19 07:56:28 [19ms] [Warning] [omni.ext.plugin] [ext: omni.kit.sequencer.core-103.4.1+105.0] Built using kit version: 105.0. Current version: 105.1. It is considered compatible, but building with a newer version is recommended.
2023-12-19 07:56:28 [20ms] [Warning] [omni.ext.plugin] [ext: omni.kit.sequencer.usd-103.4.2+105.0] Built using kit version: 105.0. Current version: 105.1. It is considered compatible, but building with a newer version is recommended.
2023-12-19 07:56:28 [23ms] [Warning] [omni.ext.plugin] [ext: omni.kit.widget.timeline-105.0.1+105.0] Built using kit version: 105.0. Current version: 105.1. It is considered compatible, but building with a newer version is recommended.
2023-12-19 07:56:28 [23ms] [Warning] [omni.ext.plugin] [ext: omni.kit.window.sequencer-103.4.2-dev.3+105.0] Built using kit version: 105.0. Current version: 105.1. It is considered compatible, but building with a newer version is recommended.
2023-12-19 07:56:28 [26ms] [Warning] [omni.ext.plugin] [ext: omni.paint.brush.attributes-1.3.1+105.0] Built using kit version: 105.0. Current version: 105.1. It is considered compatible, but building with a newer version is recommended.
2023-12-19 07:56:28 [28ms] [Warning] [omni.ext.plugin] [ext: omni.usd.schema.sequence-2.3.0+105.0.lx64.r.cp310] Built using kit version: 105.0. Current version: 105.1. It is considered compatible, but building with a newer version is recommended.
[0.061s] [ext: omni.kit.async_engine-0.0.0] startup
[0.463s] [ext: omni.assets.plugins-0.0.0] startup
[0.466s] [ext: omni.stats-0.0.0] startup
[0.467s] [ext: omni.client-1.0.1] startup
[0.478s] [ext: omni.gpu_foundation-0.0.0] startup
[0.496s] [ext: omni.rtx.shadercache.vulkan-1.0.0] startup
[0.498s] [ext: carb.windowing.plugins-1.0.0] startup
2023-12-19 07:56:28 [481ms] [Warning] [carb.windowing-glfw.plugin] GLFW initialization failed.
2023-12-19 07:56:28 [481ms] [Warning] [carb] Failed to startup plugin carb.windowing-glfw.plugin (interfaces: [carb::windowing::IGLContext v1.0],[carb::windowing::IWindowing v1.4]) (impl: carb.windowing-glfw.plugin)
[0.499s] [ext: omni.kit.renderer.init-0.0.0] startup
2023-12-19 07:56:28 [506ms] [Warning] [omni.platforminfo.plugin] failed to open the default display.  Can't verify X Server version.

|---------------------------------------------------------------------------------------------|
| Driver Version: 535.129.03    | Graphics API: Vulkan
|=============================================================================================|
| GPU | Name                             | Active | LDA | GPU Memory | Vendor-ID | LUID       |
|     |                                  |        |     |            | Device-ID | UUID       |
|     |                                  |        |     |            | Bus-ID    |            |
|---------------------------------------------------------------------------------------------|
| 0   | NVIDIA GeForce RTX 4060 Laptop.. | Yes: 0 |     | 8188    MB | 10de      | 0          |
|     |                                  |        |     |            | 28a0      | 09391f11.. |
|     |                                  |        |     |            | 1         |            |
|=============================================================================================|
| OS: 22.04.3 LTS (Jammy Jellyfish) ubuntu, Version: 22.04.3, Kernel: 6.0.12-76060012-generic
| Processor: 13th Gen Intel(R) Core(TM) i7-13700H | Cores: 14 | Logical: 28
|---------------------------------------------------------------------------------------------|
| Total Memory (MB): 15682 | Free Memory: 6238
| Total Page/Swap (MB): 39746 | Free Page/Swap: 39699
|---------------------------------------------------------------------------------------------|
[2.688s] [ext: omni.kit.pipapi-0.0.0] startup
[2.691s] [ext: omni.kit.pip_archive-0.0.0] startup
[2.691s] [ext: omni.pip.compute-1.2.0] startup
[2.692s] [ext: omni.pip.torch-2_0_1-2.0.2] startup
[2.731s] [ext: omni.pip.cloud-1.0.1] startup
[2.748s] [ext: omni.isaac.core_archive-2.2.1] startup
[2.748s] [ext: omni.isaac.ml_archive-1.1.3] startup
[2.749s] [ext: omni.kit.telemetry-0.5.0] startup
[2.783s] [ext: omni.mtlx-0.1.0] startup
[2.785s] [ext: omni.usd.config-1.0.3] startup
[2.790s] [ext: omni.gpucompute.plugins-0.0.0] startup
[2.792s] [ext: omni.usd.libs-1.0.0] startup
[2.960s] [ext: omni.kit.loop-isaac-1.1.0] startup
[2.961s] [ext: omni.kit.test-0.0.0] startup
[2.962s] [ext: omni.usd.schema.physics-0.0.0] startup
[3.172s] [ext: omni.usd.schema.physx-0.0.0] startup
[3.218s] [ext: omni.usd.schema.anim-0.0.0] startup
[3.264s] [ext: omni.usd.schema.audio-0.0.0] startup
[3.272s] [ext: omni.usd.schema.geospatial-0.0.0] startup
[3.277s] [ext: omni.usd.schema.semantics-0.0.0] startup
[3.284s] [ext: omni.usd.schema.omniscripting-1.0.0] startup
[3.294s] [ext: omni.usd.schema.omnigraph-1.0.0] startup
[3.304s] [ext: omni.kvdb-0.0.0] startup
[3.307s] [ext: omni.usd_resolver-1.0.0] startup
2023-12-19 07:56:31 [3,289ms] [Warning] [omni.ext.plugin] [ext: omni.usd_resolver-1.0.0] Native plugin search dir contains several files: /isaac-sim/kit/exts/omni.usd_resolver/bin . It makes dir walk slow, replace with the exact path to the plugin instead of a wildcard.
[3.316s] [ext: omni.localcache-0.0.0] startup
[3.319s] [ext: omni.usd.core-1.1.7] startup
[3.327s] [ext: omni.physx.foundation-105.1.10-5.1] startup
[3.328s] [ext: usdrt.scenegraph-7.2.27] startup
[3.394s] [ext: omni.hydra.scene_delegate-0.3.2] startup
[3.407s] [ext: omni.resourcemonitor-105.0.0] startup
[3.409s] [ext: omni.activity.core-1.0.1] startup
[3.412s] [ext: omni.appwindow-1.1.5] startup
2023-12-19 07:56:31 [3,396ms] [Warning] [carb.windowing-glfw.plugin] GLFW initialization failed.
2023-12-19 07:56:31 [3,396ms] [Warning] [carb] Failed to startup plugin carb.windowing-glfw.plugin (interfaces: [carb::windowing::IGLContext v1.0],[carb::windowing::IWindowing v1.4]) (impl: carb.windowing-glfw.plugin)
[3.416s] [ext: omni.hydra.usdrt_delegate-7.2.27] startup
[3.440s] [ext: omni.kit.renderer.core-0.0.0] startup
2023-12-19 07:56:31 [3,425ms] [Warning] [carb.windowing-glfw.plugin] GLFW initialization failed.
2023-12-19 07:56:31 [3,425ms] [Warning] [carb] Failed to startup plugin carb.windowing-glfw.plugin (interfaces: [carb::windowing::IGLContext v1.0],[carb::windowing::IWindowing v1.4]) (impl: carb.windowing-glfw.plugin)
2023-12-19 07:56:31 [3,432ms] [Warning] [carb.windowing-glfw.plugin] GLFW initialization failed.
2023-12-19 07:56:31 [3,432ms] [Warning] [carb] Failed to startup plugin carb.windowing-glfw.plugin (interfaces: [carb::windowing::IGLContext v1.0],[carb::windowing::IWindowing v1.4]) (impl: carb.windowing-glfw.plugin)
[3.452s] [ext: omni.usdphysics-105.1.10-5.1] startup
[3.457s] [ext: omni.kit.renderer.capture-0.0.0] startup
[3.460s] [ext: omni.kit.numpy.common-0.1.2] startup
[3.462s] [ext: omni.kit.renderer.imgui-0.0.0] startup
2023-12-19 07:56:31 [3,448ms] [Warning] [carb.windowing-glfw.plugin] GLFW initialization failed.
2023-12-19 07:56:31 [3,448ms] [Warning] [carb] Failed to startup plugin carb.windowing-glfw.plugin (interfaces: [carb::windowing::IGLContext v1.0],[carb::windowing::IWindowing v1.4]) (impl: carb.windowing-glfw.plugin)
2023-12-19 07:56:31 [3,450ms] [Warning] [carb.windowing-glfw.plugin] GLFW initialization failed.
2023-12-19 07:56:31 [3,450ms] [Warning] [carb] Failed to startup plugin carb.windowing-glfw.plugin (interfaces: [carb::windowing::IGLContext v1.0],[carb::windowing::IWindowing v1.4]) (impl: carb.windowing-glfw.plugin)
2023-12-19 07:56:31 [3,451ms] [Warning] [carb.windowing-glfw.plugin] GLFW initialization failed.
2023-12-19 07:56:31 [3,451ms] [Warning] [carb] Failed to startup plugin carb.windowing-glfw.plugin (interfaces: [carb::windowing::IGLContext v1.0],[carb::windowing::IWindowing v1.4]) (impl: carb.windowing-glfw.plugin)
[3.556s] [ext: omni.kit.actions.core-1.0.0] startup
[3.560s] [ext: omni.ui-2.18.1] startup
[3.576s] [ext: omni.graph.exec-0.3.0] startup
[3.577s] [ext: omni.kit.commands-1.4.6] startup
[3.583s] [ext: omni.kit.window.popup_dialog-2.0.23] startup
[3.589s] [ext: carb.audio-0.1.0] startup
[3.593s] [ext: omni.kit.exec.core-0.5.0] startup
[3.596s] [ext: omni.timeline-1.0.9] startup
[3.598s] [ext: omni.kit.widget.nucleus_connector-1.1.4] startup
[3.601s] [ext: omni.kit.audiodeviceenum-1.0.1] startup
[3.603s] [ext: omni.convexdecomposition-105.1.10-5.1] startup
[3.611s] [ext: omni.usd-1.10.18] startup
[3.684s] [ext: omni.physx.cooking-105.1.10-5.1] startup
[3.698s] [ext: omni.iray.libs-0.0.0] startup
[3.706s] [ext: omni.physx-105.1.10-5.1] startup
2023-12-19 07:56:31 [3,711ms] [Warning] [omni.stageupdate.plugin] Deprecated: direct use of IStageUpdate callbacks is deprecated. Use IStageUpdate::getStageUpdate instead.
[3.729s] [ext: omni.mdl.neuraylib-0.2.0] startup
[3.732s] [ext: omni.isaac.dynamic_control-1.2.5] startup
[3.747s] [ext: omni.kit.widget.versioning-1.4.5] startup
[3.750s] [ext: omni.mdl-0.1.2] startup
[3.785s] [ext: omni.kit.helper.file_utils-0.1.6] startup
[3.818s] [ext: omni.kit.widget.nucleus_info-1.0.1] startup
[3.819s] [ext: omni.kit.widget.path_field-2.0.8] startup
[3.820s] [ext: omni.kit.widget.filebrowser-2.3.35] startup
[3.824s] [ext: omni.kit.search_core-1.0.5] startup
[3.825s] [ext: omni.kit.widget.browser_bar-2.0.8] startup
[3.826s] [ext: omni.kit.widget.search_delegate-1.0.4] startup
[3.827s] [ext: omni.ui.scene-1.6.15] startup
[3.834s] [ext: omni.kit.notification_manager-1.0.6] startup
[3.836s] [ext: omni.kit.clipboard-1.0.3] startup
[3.837s] [ext: omni.isaac.kit-1.4.5] startup
[3.837s] [ext: omni.kit.window.filepicker-2.10.7] startup
[3.845s] [ext: omni.kit.usd.layers-2.1.23] startup
[3.858s] [ext: omni.physics.tensors-0.1.0] startup
[3.869s] [ext: omni.kit.window.file_importer-1.0.21] startup
[3.871s] [ext: omni.kit.menu.utils-1.5.6] startup
[3.884s] [ext: omni.physx.tensors-0.1.0] startup
[3.892s] [ext: omni.warp.core-0.10.1] startup
Warp 0.10.1 initialized:
   CUDA Toolkit: 11.5, Driver: 12.2
   Devices:
     "cpu"    | x86_64
     "cuda:0" | NVIDIA GeForce RTX 4060 Laptop GPU (sm_89)
   Kernel cache: /root/.cache/warp/0.10.1
[4.056s] [ext: omni.kit.material.library-1.3.40] startup
[4.061s] [ext: omni.isaac.version-1.0.3] startup
[4.061s] [ext: omni.kit.window.title-1.1.3] startup
2023-12-19 07:56:32 [4,045ms] [Warning] [carb.windowing-glfw.plugin] GLFW initialization failed.
2023-12-19 07:56:32 [4,045ms] [Warning] [carb] Failed to startup plugin carb.windowing-glfw.plugin (interfaces: [carb::windowing::IGLContext v1.0],[carb::windowing::IWindowing v1.4]) (impl: carb.windowing-glfw.plugin)
2023-12-19 07:56:32 [4,046ms] [Warning] [carb.windowing-glfw.plugin] GLFW initialization failed.
2023-12-19 07:56:32 [4,046ms] [Warning] [carb] Failed to startup plugin carb.windowing-glfw.plugin (interfaces: [carb::windowing::IGLContext v1.0],[carb::windowing::IWindowing v1.4]) (impl: carb.windowing-glfw.plugin)
[4.065s] [ext: omni.isaac.core-2.10.2] startup
[4.391s] [ext: omni.isaac.cloner-0.7.1] startup
[4.391s] [ext: omni.physx.fabric-105.1.10-5.1] startup
[4.400s] [ext: omni.kit.mainwindow-1.0.1] startup
[4.401s] [ext: omni.isaac.gym-0.8.5] startup
[4.402s] [ext: omni.isaac.sim.python.gym.headless-2023.1.0] startup
[4.403s] Simulation App Starting
[4.573s] app ready
[5.135s] Simulation App Startup Complete
task_name: Cartpole
experiment: 
num_envs: 
seed: 42
torch_deterministic: False
max_iterations: 
physics_engine: physx
pipeline: gpu
sim_device: gpu
device_id: 0
rl_device: cuda:0
multi_gpu: False
num_threads: 4
solver_type: 1
test: False
checkpoint: 
evaluation: False
headless: True
enable_livestream: False
mt_timeout: 90
wandb_activate: False
wandb_group: 
wandb_name: Cartpole
wandb_entity: 
wandb_project: omniisaacgymenvs
kit_app: 
warp: False
task: 
    name: Cartpole
    physics_engine: physx
    env: 
        numEnvs: 8
        envSpacing: 4.0
        resetDist: 3.0
        maxEffort: 400.0
        clipObservations: 5.0
        clipActions: 1.0
        controlFrequencyInv: 2
    sim: 
        dt: 0.0083
        use_gpu_pipeline: True
        gravity: [0.0, 0.0, -9.81]
        add_ground_plane: True
        add_distant_light: False
        use_fabric: True
        enable_scene_query_support: False
        disable_contact_processing: False
        enable_cameras: False
        default_physics_material: 
            static_friction: 1.0
            dynamic_friction: 1.0
            restitution: 0.0
        physx: 
            worker_thread_count: 4
            solver_type: 1
            use_gpu: True
            solver_position_iteration_count: 4
            solver_velocity_iteration_count: 0
            contact_offset: 0.02
            rest_offset: 0.001
            bounce_threshold_velocity: 0.2
            friction_offset_threshold: 0.04
            friction_correlation_distance: 0.025
            enable_sleeping: True
            enable_stabilization: True
            max_depenetration_velocity: 100.0
            gpu_max_rigid_contact_count: 524288
            gpu_max_rigid_patch_count: 81920
            gpu_found_lost_pairs_capacity: 1024
            gpu_found_lost_aggregate_pairs_capacity: 262144
            gpu_total_aggregate_pairs_capacity: 1024
            gpu_max_soft_body_contacts: 1048576
            gpu_max_particle_contacts: 1048576
            gpu_heap_capacity: 67108864
            gpu_temp_buffer_capacity: 16777216
            gpu_max_num_partitions: 8
        Cartpole: 
            override_usd_defaults: False
            enable_self_collisions: False
            enable_gyroscopic_forces: True
            solver_position_iteration_count: 4
            solver_velocity_iteration_count: 0
            sleep_threshold: 0.005
            stabilization_threshold: 0.001
            density: -1
            max_depenetration_velocity: 100.0
            contact_offset: 0.02
            rest_offset: 0.001
train: 
    params: 
        seed: 42
        algo: 
            name: a2c_continuous
        model: 
            name: continuous_a2c_logstd
        network: 
            name: actor_critic
            separate: False
            space: 
                continuous: 
                    mu_activation: None
                    sigma_activation: None
                    mu_init: 
                        name: default
                    sigma_init: 
                        name: const_initializer
                        val: 0
                    fixed_sigma: True
            mlp: 
                units: [32, 32]
                activation: elu
                initializer: 
                    name: default
                regularizer: 
                    name: None
        load_checkpoint: False
        load_path: 
        config: 
            name: Cartpole
            full_experiment_name: Cartpole
            device: cuda:0
            device_name: cuda:0
            env_name: rlgpu
            multi_gpu: False
            ppo: True
            mixed_precision: False
            normalize_input: True
            normalize_value: True
            num_actors: 8
            reward_shaper: 
                scale_value: 0.1
            normalize_advantage: True
            gamma: 0.99
            tau: 0.95
            learning_rate: 0.0003
            lr_schedule: adaptive
            kl_threshold: 0.008
            score_to_win: 20000
            max_epochs: 100
            save_best_after: 50
            save_frequency: 25
            grad_norm: 1.0
            entropy_coef: 0.0
            truncate_grads: True
            e_clip: 0.2
            horizon_length: 16
            minibatch_size: 8192
            mini_epochs: 8
            critic_coef: 4
            clip_value: True
            seq_length: 4
            bounds_loss_coef: 0.0001
Setting seed: 42
Sim params does not have attribute:  physx
Sim params does not have attribute:  Cartpole
Pipeline:  GPU
Pipeline Device:  cuda:0
Sim Device:  GPU
Task Device: cuda:0
RL device:  cuda:0
2023-12-19 07:56:33 [5,327ms] [Warning] [omni.isaac.core.utils.viewports] omni.kit.viewport.utility needs to be enabled before using this function
2023-12-19 07:56:43 [15,382ms] [Warning] [omni.client.python] Detected a blocking function. This will cause hitches or hangs in the UI. Please switch to the async version:
  File "/workspace/omniisaacgymenvs/omniisaacgymenvs/scripts/rlgames_train.py", line 150, in <module>
  File "/isaac-sim/kit/python/lib/python3.10/site-packages/hydra/main.py", line 94, in decorated_main
  File "/isaac-sim/kit/python/lib/python3.10/site-packages/hydra/_internal/utils.py", line 394, in _run_hydra
  File "/isaac-sim/kit/python/lib/python3.10/site-packages/hydra/_internal/utils.py", line 457, in _run_app
  File "/isaac-sim/kit/python/lib/python3.10/site-packages/hydra/_internal/utils.py", line 220, in run_and_report
  File "/isaac-sim/kit/python/lib/python3.10/site-packages/hydra/_internal/utils.py", line 458, in <lambda>
  File "/isaac-sim/kit/python/lib/python3.10/site-packages/hydra/_internal/hydra.py", line 119, in run
  File "/isaac-sim/kit/python/lib/python3.10/site-packages/hydra/core/utils.py", line 186, in run_job
  File "/workspace/omniisaacgymenvs/omniisaacgymenvs/scripts/rlgames_train.py", line 121, in parse_hydra_configs
  File "/workspace/omniisaacgymenvs/omniisaacgymenvs/utils/task_util.py", line 103, in initialize_task
  File "/workspace/omniisaacgymenvs/omniisaacgymenvs/envs/vec_env_rlgames.py", line 47, in set_task
  File "/isaac-sim/exts/omni.isaac.gym/omni/isaac/gym/vec_env/vec_env_base.py", line 126, in set_task
  File "/isaac-sim/exts/omni.isaac.core/omni/isaac/core/world/world.py", line 276, in reset
  File "/workspace/omniisaacgymenvs/omniisaacgymenvs/tasks/cartpole.py", line 65, in set_up_scene
  File "/workspace/omniisaacgymenvs/omniisaacgymenvs/tasks/cartpole.py", line 83, in get_cartpole
  File "/workspace/omniisaacgymenvs/omniisaacgymenvs/robots/articulations/cartpole.py", line 54, in __init__
  File "/isaac-sim/exts/omni.isaac.core/omni/isaac/core/utils/nucleus.py", line 504, in get_assets_root_path
  File "/isaac-sim/exts/omni.isaac.core/omni/isaac/core/utils/nucleus.py", line 191, in check_server
  File "/isaac-sim/kit/extscore/omni.client/omni/client/__init__.py", line 161, in stat

2023-12-19 07:58:44 [136,758ms] [Warning] [omni.isaac.core.utils.nucleus] /persistent/app/omniverse/mountedDrives setting not found
2023-12-19 07:58:46 [138,239ms] [Warning] [omni.isaac.core.utils.nucleus] /persistent/app/omniverse/mountedDrives setting not found
~*~*~*
~*~*~* Direct GPU Helper:
~*~*~*   8 articulations, maxLinks=3
~*~*~*
self.seed = 42
Started to train
Exact experiment name requested from command line: Cartpole
Box(-1.0, 1.0, (1,), float32) Box(-inf, inf, (4,), float32)
seq_length: 4
current training device: cuda:0
2023-12-19 07:58:46 [138,810ms] [Error] [omni.kit.app._impl] [py stderr]: Error executing job with overrides: ['task=Cartpole', 'headless=True']

Error executing job with overrides: ['task=Cartpole', 'headless=True']
2023-12-19 07:58:46 [138,810ms] [Error] [omni.kit.app._impl] [py stderr]: Traceback (most recent call last):

Traceback (most recent call last):
2023-12-19 07:58:46 [138,810ms] [Error] [omni.kit.app._impl] [py stderr]:   File "/workspace/omniisaacgymenvs/omniisaacgymenvs/scripts/rlgames_train.py", line 150, in <module>

  File "/workspace/omniisaacgymenvs/omniisaacgymenvs/scripts/rlgames_train.py", line 150, in <module>
2023-12-19 07:58:46 [138,811ms] [Error] [omni.kit.app._impl] [py stderr]:     
2023-12-19 07:58:46 [138,811ms] [Error] [omni.kit.app._impl] [py stderr]: parse_hydra_configs()
2023-12-19 07:58:46 [138,811ms] [Error] [omni.kit.app._impl] [py stderr]: 

    parse_hydra_configs()
2023-12-19 07:58:46 [138,811ms] [Error] [omni.kit.app._impl] [py stderr]:   File "/isaac-sim/kit/python/lib/python3.10/site-packages/hydra/main.py", line 94, in decorated_main

  File "/isaac-sim/kit/python/lib/python3.10/site-packages/hydra/main.py", line 94, in decorated_main
2023-12-19 07:58:46 [138,811ms] [Error] [omni.kit.app._impl] [py stderr]:     
2023-12-19 07:58:46 [138,811ms] [Error] [omni.kit.app._impl] [py stderr]: _run_hydra(
2023-12-19 07:58:46 [138,811ms] [Error] [omni.kit.app._impl] [py stderr]: 

    _run_hydra(
2023-12-19 07:58:46 [138,811ms] [Error] [omni.kit.app._impl] [py stderr]:   File "/isaac-sim/kit/python/lib/python3.10/site-packages/hydra/_internal/utils.py", line 394, in _run_hydra

  File "/isaac-sim/kit/python/lib/python3.10/site-packages/hydra/_internal/utils.py", line 394, in _run_hydra
2023-12-19 07:58:46 [138,811ms] [Error] [omni.kit.app._impl] [py stderr]:     
2023-12-19 07:58:46 [138,811ms] [Error] [omni.kit.app._impl] [py stderr]: _run_app(
2023-12-19 07:58:46 [138,811ms] [Error] [omni.kit.app._impl] [py stderr]: 

    _run_app(
2023-12-19 07:58:46 [138,811ms] [Error] [omni.kit.app._impl] [py stderr]:   File "/isaac-sim/kit/python/lib/python3.10/site-packages/hydra/_internal/utils.py", line 457, in _run_app

  File "/isaac-sim/kit/python/lib/python3.10/site-packages/hydra/_internal/utils.py", line 457, in _run_app
2023-12-19 07:58:46 [138,811ms] [Error] [omni.kit.app._impl] [py stderr]:     
2023-12-19 07:58:46 [138,811ms] [Error] [omni.kit.app._impl] [py stderr]: run_and_report(
2023-12-19 07:58:46 [138,811ms] [Error] [omni.kit.app._impl] [py stderr]: 

    run_and_report(
2023-12-19 07:58:46 [138,811ms] [Error] [omni.kit.app._impl] [py stderr]:   File "/isaac-sim/kit/python/lib/python3.10/site-packages/hydra/_internal/utils.py", line 223, in run_and_report

  File "/isaac-sim/kit/python/lib/python3.10/site-packages/hydra/_internal/utils.py", line 223, in run_and_report
2023-12-19 07:58:46 [138,812ms] [Error] [omni.kit.app._impl] [py stderr]:     
2023-12-19 07:58:46 [138,812ms] [Error] [omni.kit.app._impl] [py stderr]: raise ex
2023-12-19 07:58:46 [138,812ms] [Error] [omni.kit.app._impl] [py stderr]: 

    raise ex
2023-12-19 07:58:46 [138,812ms] [Error] [omni.kit.app._impl] [py stderr]:   File "/isaac-sim/kit/python/lib/python3.10/site-packages/hydra/_internal/utils.py", line 220, in run_and_report

  File "/isaac-sim/kit/python/lib/python3.10/site-packages/hydra/_internal/utils.py", line 220, in run_and_report
2023-12-19 07:58:46 [138,812ms] [Error] [omni.kit.app._impl] [py stderr]:     
2023-12-19 07:58:46 [138,812ms] [Error] [omni.kit.app._impl] [py stderr]: return func()
2023-12-19 07:58:46 [138,812ms] [Error] [omni.kit.app._impl] [py stderr]: 

    return func()
2023-12-19 07:58:46 [138,812ms] [Error] [omni.kit.app._impl] [py stderr]:   File "/isaac-sim/kit/python/lib/python3.10/site-packages/hydra/_internal/utils.py", line 458, in <lambda>

  File "/isaac-sim/kit/python/lib/python3.10/site-packages/hydra/_internal/utils.py", line 458, in <lambda>
2023-12-19 07:58:46 [138,812ms] [Error] [omni.kit.app._impl] [py stderr]:     
2023-12-19 07:58:46 [138,812ms] [Error] [omni.kit.app._impl] [py stderr]: lambda: hydra.run(
2023-12-19 07:58:46 [138,812ms] [Error] [omni.kit.app._impl] [py stderr]: 

    lambda: hydra.run(
2023-12-19 07:58:46 [138,812ms] [Error] [omni.kit.app._impl] [py stderr]:   File "/isaac-sim/kit/python/lib/python3.10/site-packages/hydra/_internal/hydra.py", line 132, in run

  File "/isaac-sim/kit/python/lib/python3.10/site-packages/hydra/_internal/hydra.py", line 132, in run
2023-12-19 07:58:46 [138,812ms] [Error] [omni.kit.app._impl] [py stderr]:     
2023-12-19 07:58:46 [138,812ms] [Error] [omni.kit.app._impl] [py stderr]: _ = ret.return_value
2023-12-19 07:58:46 [138,812ms] [Error] [omni.kit.app._impl] [py stderr]: 

    _ = ret.return_value
2023-12-19 07:58:46 [138,812ms] [Error] [omni.kit.app._impl] [py stderr]:   File "/isaac-sim/kit/python/lib/python3.10/site-packages/hydra/core/utils.py", line 260, in return_value

  File "/isaac-sim/kit/python/lib/python3.10/site-packages/hydra/core/utils.py", line 260, in return_value
2023-12-19 07:58:46 [138,812ms] [Error] [omni.kit.app._impl] [py stderr]:     
2023-12-19 07:58:46 [138,812ms] [Error] [omni.kit.app._impl] [py stderr]: raise self._return_value
2023-12-19 07:58:46 [138,812ms] [Error] [omni.kit.app._impl] [py stderr]: 

    raise self._return_value
2023-12-19 07:58:46 [138,812ms] [Error] [omni.kit.app._impl] [py stderr]:   File "/isaac-sim/kit/python/lib/python3.10/site-packages/hydra/core/utils.py", line 186, in run_job

  File "/isaac-sim/kit/python/lib/python3.10/site-packages/hydra/core/utils.py", line 186, in run_job
2023-12-19 07:58:46 [138,813ms] [Error] [omni.kit.app._impl] [py stderr]:     
2023-12-19 07:58:46 [138,813ms] [Error] [omni.kit.app._impl] [py stderr]: ret.return_value = task_function(task_cfg)
2023-12-19 07:58:46 [138,813ms] [Error] [omni.kit.app._impl] [py stderr]: 

    ret.return_value = task_function(task_cfg)
2023-12-19 07:58:46 [138,813ms] [Error] [omni.kit.app._impl] [py stderr]:   File "/workspace/omniisaacgymenvs/omniisaacgymenvs/scripts/rlgames_train.py", line 142, in parse_hydra_configs

  File "/workspace/omniisaacgymenvs/omniisaacgymenvs/scripts/rlgames_train.py", line 142, in parse_hydra_configs
2023-12-19 07:58:46 [138,813ms] [Error] [omni.kit.app._impl] [py stderr]:     
2023-12-19 07:58:46 [138,813ms] [Error] [omni.kit.app._impl] [py stderr]: rlg_trainer.run()
2023-12-19 07:58:46 [138,813ms] [Error] [omni.kit.app._impl] [py stderr]: 

    rlg_trainer.run()
2023-12-19 07:58:46 [138,813ms] [Error] [omni.kit.app._impl] [py stderr]:   File "/workspace/omniisaacgymenvs/omniisaacgymenvs/scripts/rlgames_train.py", line 74, in run

  File "/workspace/omniisaacgymenvs/omniisaacgymenvs/scripts/rlgames_train.py", line 74, in run
2023-12-19 07:58:46 [138,813ms] [Error] [omni.kit.app._impl] [py stderr]:     
2023-12-19 07:58:46 [138,813ms] [Error] [omni.kit.app._impl] [py stderr]: runner.run(
2023-12-19 07:58:46 [138,813ms] [Error] [omni.kit.app._impl] [py stderr]: 

    runner.run(
2023-12-19 07:58:46 [138,813ms] [Error] [omni.kit.app._impl] [py stderr]:   File "/isaac-sim/kit/python/lib/python3.10/site-packages/rl_games/torch_runner.py", line 133, in run

  File "/isaac-sim/kit/python/lib/python3.10/site-packages/rl_games/torch_runner.py", line 133, in run
2023-12-19 07:58:46 [138,813ms] [Error] [omni.kit.app._impl] [py stderr]:     
2023-12-19 07:58:46 [138,813ms] [Error] [omni.kit.app._impl] [py stderr]: self.run_train(args)
2023-12-19 07:58:46 [138,813ms] [Error] [omni.kit.app._impl] [py stderr]: 

    self.run_train(args)
2023-12-19 07:58:46 [138,813ms] [Error] [omni.kit.app._impl] [py stderr]:   File "/isaac-sim/kit/python/lib/python3.10/site-packages/rl_games/torch_runner.py", line 113, in run_train

  File "/isaac-sim/kit/python/lib/python3.10/site-packages/rl_games/torch_runner.py", line 113, in run_train
2023-12-19 07:58:46 [138,813ms] [Error] [omni.kit.app._impl] [py stderr]:     
2023-12-19 07:58:46 [138,813ms] [Error] [omni.kit.app._impl] [py stderr]: agent = self.algo_factory.create(self.algo_name, base_name='run', params=self.params)
2023-12-19 07:58:46 [138,813ms] [Error] [omni.kit.app._impl] [py stderr]: 

    agent = self.algo_factory.create(self.algo_name, base_name='run', params=self.params)
2023-12-19 07:58:46 [138,813ms] [Error] [omni.kit.app._impl] [py stderr]:   File "/isaac-sim/kit/python/lib/python3.10/site-packages/rl_games/common/object_factory.py", line 15, in create

  File "/isaac-sim/kit/python/lib/python3.10/site-packages/rl_games/common/object_factory.py", line 15, in create
2023-12-19 07:58:46 [138,813ms] [Error] [omni.kit.app._impl] [py stderr]:     
2023-12-19 07:58:46 [138,813ms] [Error] [omni.kit.app._impl] [py stderr]: return builder(**kwargs)
2023-12-19 07:58:46 [138,813ms] [Error] [omni.kit.app._impl] [py stderr]: 

    return builder(**kwargs)
2023-12-19 07:58:46 [138,814ms] [Error] [omni.kit.app._impl] [py stderr]:   File "/isaac-sim/kit/python/lib/python3.10/site-packages/rl_games/torch_runner.py", line 37, in <lambda>

  File "/isaac-sim/kit/python/lib/python3.10/site-packages/rl_games/torch_runner.py", line 37, in <lambda>
2023-12-19 07:58:46 [138,814ms] [Error] [omni.kit.app._impl] [py stderr]:     
2023-12-19 07:58:46 [138,814ms] [Error] [omni.kit.app._impl] [py stderr]: self.algo_factory.register_builder('a2c_continuous', lambda **kwargs : a2c_continuous.A2CAgent(**kwargs))
2023-12-19 07:58:46 [138,814ms] [Error] [omni.kit.app._impl] [py stderr]: 

    self.algo_factory.register_builder('a2c_continuous', lambda **kwargs : a2c_continuous.A2CAgent(**kwargs))
2023-12-19 07:58:46 [138,814ms] [Error] [omni.kit.app._impl] [py stderr]:   File "/isaac-sim/kit/python/lib/python3.10/site-packages/rl_games/algos_torch/a2c_continuous.py", line 15, in __init__

  File "/isaac-sim/kit/python/lib/python3.10/site-packages/rl_games/algos_torch/a2c_continuous.py", line 15, in __init__
2023-12-19 07:58:46 [138,814ms] [Error] [omni.kit.app._impl] [py stderr]:     
2023-12-19 07:58:46 [138,814ms] [Error] [omni.kit.app._impl] [py stderr]: a2c_common.ContinuousA2CBase.__init__(self, base_name, params)
2023-12-19 07:58:46 [138,814ms] [Error] [omni.kit.app._impl] [py stderr]: 

    a2c_common.ContinuousA2CBase.__init__(self, base_name, params)
2023-12-19 07:58:46 [138,814ms] [Error] [omni.kit.app._impl] [py stderr]:   File "/isaac-sim/kit/python/lib/python3.10/site-packages/rl_games/common/a2c_common.py", line 1143, in __init__

  File "/isaac-sim/kit/python/lib/python3.10/site-packages/rl_games/common/a2c_common.py", line 1143, in __init__
2023-12-19 07:58:46 [138,814ms] [Error] [omni.kit.app._impl] [py stderr]:     
2023-12-19 07:58:46 [138,814ms] [Error] [omni.kit.app._impl] [py stderr]: A2CBase.__init__(self, base_name, params)
2023-12-19 07:58:46 [138,814ms] [Error] [omni.kit.app._impl] [py stderr]: 

    A2CBase.__init__(self, base_name, params)
2023-12-19 07:58:46 [138,814ms] [Error] [omni.kit.app._impl] [py stderr]:   File "/isaac-sim/kit/python/lib/python3.10/site-packages/rl_games/common/a2c_common.py", line 249, in __init__

  File "/isaac-sim/kit/python/lib/python3.10/site-packages/rl_games/common/a2c_common.py", line 249, in __init__
2023-12-19 07:58:46 [138,814ms] [Error] [omni.kit.app._impl] [py stderr]:     
2023-12-19 07:58:46 [138,814ms] [Error] [omni.kit.app._impl] [py stderr]: assert(self.batch_size % self.minibatch_size == 0)
2023-12-19 07:58:46 [138,814ms] [Error] [omni.kit.app._impl] [py stderr]: 

    assert(self.batch_size % self.minibatch_size == 0)
2023-12-19 07:58:46 [138,814ms] [Error] [omni.kit.app._impl] [py stderr]: AssertionError
2023-12-19 07:58:46 [138,814ms] [Error] [omni.kit.app._impl] [py stderr]: 
2023-12-19 07:58:46 [138,814ms] [Error] [omni.kit.app._impl] [py stderr]: 

AssertionError
2023-12-19 07:58:47 [139,029ms] [Warning] [omni.stageupdate.plugin] Deprecated: direct use of IStageUpdate callbacks is deprecated. Use IStageUpdate::getStageUpdate instead.
2023-12-19 07:58:47 [139,029ms] [Warning] [carb.audio.context] 1 contexts were leaked
2023-12-19 07:58:47 [139,039ms] [Warning] [carb] Recursive unloadAllPlugins() detected!
2023-12-19 07:58:47 [139,049ms] [Warning] [omni.core.ITypeFactory] Module /isaac-sim/kit/exts/omni.activity.core/bin/libomni.activity.core.plugin.so remained loaded after unload request.
Toni-SM commented 10 months ago

Hi @Cdfghglz

Now the problem is the RL training configuration.

2023-12-19 07:58:46 [138,814ms] [Error] [omni.kit.app._impl] [py stderr]: assert(self.batch_size % self.minibatch_size == 0)

Note: your num_envs is 8

Check the first point in troubleshoot: RL Training

Cdfghglz commented 10 months ago

Thanks a lot @Toni-SM , changed the num_envs to 512 the example works nicely.

Would it be a worthy improvement to change the defaults to something that works out of the box or am I missing something about the config?