Closed Robokan closed 1 year ago
kellyguo11
commented
1 year ago Hi there, the error seems to suggest that one of the Isaac Sim instances was not launched on the correct GPU device. This was a new feature introduced in the latest Isaac Sim 2022.2.1 release, could you try updating to this version of Isaac Sim?
Robokan
commented
1 year ago I updated from 2022.2.0 to 2022.2.1 and it does work! thanks.
Robokan
commented
1 year ago only thing is it is using a very small amount of the memory and resources on each GPU. Any attempt to increase the number of environments just crashes with memory errors. AnymalTerrain uses 2048 and if I double it to 4096 it does not work. Even though I am only using a fraction of the memory on each GPU.
# I am using a conda environment which has been set up properly as per instructions python -m torch.distributed.run --nnodes=1 --nproc_per_node=2 scripts/rlgames_train.py headless=True task=Ant multi_gpu=True
Any ideas? I would love to be able to use my other 4090!
stack trace:
python -m torch.distributed.run --nnodes=1 --nproc_per_node=2 scripts/rlgames_train.py headless=True task=Ant multi_gpu=True WARNING:main:
Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.
/home/bizon/anaconda3/envs/isaac-sim/lib/python3.7/site-packages/hydra/_internal/defaults_list.py:251: UserWarning: In 'config': Defaults list is missing
_self_. See https://hydra.cc/docs/upgrades/1.0_to_1.1/default_composition_order for more information warnings.warn(msg, UserWarning) /home/bizon/anaconda3/envs/isaac-sim/lib/python3.7/site-packages/hydra/_internal/defaults_list.py:412: UserWarning: In config: Invalid overriding of hydra/job_logging: Default list overrides requires 'override' keyword. See https://hydra.cc/docs/next/upgrades/1.0_to_1.1/defaults_list_override for more information.deprecation_warning(msg) /home/bizon/anaconda3/envs/isaac-sim/lib/python3.7/site-packages/hydra/_internal/defaults_list.py:251: UserWarning: In 'config': Defaults list is missing
_self_. See https://hydra.cc/docs/upgrades/1.0_to_1.1/default_composition_order for more information warnings.warn(msg, UserWarning) /home/bizon/anaconda3/envs/isaac-sim/lib/python3.7/site-packages/hydra/_internal/defaults_list.py:412: UserWarning: In config: Invalid overriding of hydra/job_logging: Default list overrides requires 'override' keyword. See https://hydra.cc/docs/next/upgrades/1.0_to_1.1/defaults_list_override for more information.deprecation_warning(msg) [Warning] [omni.isaac.kit.simulation_app] Modules: ['omniisaacgymenvs', 'omniisaacgymenvs.utils', 'omniisaacgymenvs.utils.hydra_cfg', 'omniisaacgymenvs.utils.hydra_cfg.hydra_utils', 'omniisaacgymenvs.utils.hydra_cfg.reformat', 'omniisaacgymenvs.utils.rlgames', 'omniisaacgymenvs.utils.rlgames.rlgames_utils', 'omniisaacgymenvs.utils.task_util', 'omniisaacgymenvs.utils.config_utils', 'omniisaacgymenvs.utils.config_utils.path_utils', 'omniisaacgymenvs.envs', 'omniisaacgymenvs.envs.vec_env_rlgames', 'omni', 'omni.isaac.gym', 'omni.isaac.gym.vec_env', 'omni.isaac.gym.vec_env.vec_env_base', 'omni.kit.app._impl.telemetry_helpers', 'omni.isaac.gym.vec_env.vec_env_mt'] were loaded before SimulationApp was started and might not be loaded correctly. [Warning] [omni.isaac.kit.simulation_app] Please check to make sure no extra omniverse or pxr modules are imported before the call to SimulationApp(...) Starting kit application with the following args: ['/home/bizon/.local/share/ov/pkg/isaac_sim-2022.2.0/exts/omni.isaac.kit/omni/isaac/kit/simulation_app.py', '/home/bizon/.local/share/ov/pkg/isaac_sim-2022.2.0/apps/omni.isaac.sim.python.gym.headless.kit', '--/app/tokens/exe-path=/home/bizon/.local/share/ov/pkg/isaac_sim-2022.2.0/kit', '--/persistent/app/viewport/displayOptions=3094', '--/rtx/materialDb/syncLoads=True', '--/rtx/hydra/materialSyncLoads=True--/omni.kit.plugin/syncUsdLoads=True', '--/app/renderer/resolution/width=1280', '--/app/renderer/resolution/height=720', '--/app/window/width=1440', '--/app/window/height=900', '--/renderer/multiGpu/enabled=True', '--/app/fastShutdown=True', '--ext-folder', '/home/bizon/.local/share/ov/pkg/isaac_sim-2022.2.0/exts', '--ext-folder', '/home/bizon/.local/share/ov/pkg/isaac_sim-2022.2.0/apps', '--/physics/cudaDevice=0', '--portable', '--no-window'] Passing the following args to the base kit application: ['headless=True', 'task=Ant', 'multi_gpu=True'] [Warning] [omni.isaac.kit.simulation_app] Modules: ['omniisaacgymenvs', 'omniisaacgymenvs.utils', 'omniisaacgymenvs.utils.hydra_cfg', 'omniisaacgymenvs.utils.hydra_cfg.hydra_utils', 'omniisaacgymenvs.utils.hydra_cfg.reformat', 'omniisaacgymenvs.utils.rlgames', 'omniisaacgymenvs.utils.rlgames.rlgames_utils', 'omniisaacgymenvs.utils.task_util', 'omniisaacgymenvs.utils.config_utils', 'omniisaacgymenvs.utils.config_utils.path_utils', 'omniisaacgymenvs.envs', 'omniisaacgymenvs.envs.vec_env_rlgames', 'omni', 'omni.isaac.gym', 'omni.isaac.gym.vec_env', 'omni.isaac.gym.vec_env.vec_env_base', 'omni.kit.app._impl.telemetry_helpers', 'omni.isaac.gym.vec_env.vec_env_mt'] were loaded before SimulationApp was started and might not be loaded correctly. [Warning] [omni.isaac.kit.simulation_app] Please check to make sure no extra omniverse or pxr modules are imported before the call to SimulationApp(...) Starting kit application with the following args: ['/home/bizon/.local/share/ov/pkg/isaac_sim-2022.2.0/exts/omni.isaac.kit/omni/isaac/kit/simulation_app.py', '/home/bizon/.local/share/ov/pkg/isaac_sim-2022.2.0/apps/omni.isaac.sim.python.gym.headless.kit', '--/app/tokens/exe-path=/home/bizon/.local/share/ov/pkg/isaac_sim-2022.2.0/kit', '--/persistent/app/viewport/displayOptions=3094', '--/rtx/materialDb/syncLoads=True', '--/rtx/hydra/materialSyncLoads=True--/omni.kit.plugin/syncUsdLoads=True', '--/app/renderer/resolution/width=1280', '--/app/renderer/resolution/height=720', '--/app/window/width=1440', '--/app/window/height=900', '--/renderer/multiGpu/enabled=True', '--/app/fastShutdown=True', '--ext-folder', '/home/bizon/.local/share/ov/pkg/isaac_sim-2022.2.0/exts', '--ext-folder', '/home/bizon/.local/share/ov/pkg/isaac_sim-2022.2.0/apps', '--/physics/cudaDevice=0', '--portable', '--no-window'] Passing the following args to the base kit application: ['headless=True', 'task=Ant', 'multi_gpu=True'] [Warning] [omni.kit.app.plugin] No crash reporter present, dumps uploading isn't available. [Info] [carb] Logging to file: /home/bizon/.local/share/ov/pkg/isaac_sim-2022.2.0/kit/logs/Kit/Isaac-Sim/2022.2/kit_20230504_201332.log 2023-05-05 06:13:32 [1ms] [Warning] [omni.ext.plugin] [ext: omni.drivesim.sensors.nv.radar] Extensions config 'extension.toml' doesn't exist '/home/bizon/.local/share/ov/pkg/isaac_sim-2022.2.0/exts/omni.drivesim.sensors.nv.radar' or '/home/bizon/.local/share/ov/pkg/isaac_sim-2022.2.0/exts/omni.drivesim.sensors.nv.radar/config' 2023-05-05 06:13:32 [1ms] [Warning] [omni.ext.plugin] [ext: omni.drivesim.sensors.nv.lidar] Extensions config 'extension.toml' doesn't exist '/home/bizon/.local/share/ov/pkg/isaac_sim-2022.2.0/exts/omni.drivesim.sensors.nv.lidar' or '/home/bizon/.local/share/ov/pkg/isaac_sim-2022.2.0/exts/omni.drivesim.sensors.nv.lidar/config' [Warning] [omni.kit.app.plugin] No crash reporter present, dumps uploading isn't available. [Info] [carb] Logging to file: /home/bizon/.local/share/ov/pkg/isaac_sim-2022.2.0/kit/logs/Kit/Isaac-Sim/2022.2/kit_20230504_201332.log 2023-05-05 06:13:32 [1ms] [Warning] [omni.ext.plugin] [ext: omni.drivesim.sensors.nv.lidar] Extensions config 'extension.toml' doesn't exist '/home/bizon/.local/share/ov/pkg/isaac_sim-2022.2.0/exts/omni.drivesim.sensors.nv.lidar' or '/home/bizon/.local/share/ov/pkg/isaac_sim-2022.2.0/exts/omni.drivesim.sensors.nv.lidar/config' 2023-05-05 06:13:32 [1ms] [Warning] [omni.ext.plugin] [ext: omni.drivesim.sensors.nv.radar] Extensions config 'extension.toml' doesn't exist '/home/bizon/.local/share/ov/pkg/isaac_sim-2022.2.0/exts/omni.drivesim.sensors.nv.radar' or '/home/bizon/.local/share/ov/pkg/isaac_sim-2022.2.0/exts/omni.drivesim.sensors.nv.radar/config' [0.090s] [ext: omni.stats-0.0.0] startup [0.090s] [ext: omni.stats-0.0.0] startup [0.094s] [ext: omni.rtx.shadercache-1.0.0] startup [0.094s] [ext: omni.rtx.shadercache-1.0.0] startup [0.098s] [ext: omni.assets.plugins-0.0.0] startup [0.099s] [ext: omni.gpu_foundation-0.0.0] startup [0.097s] [ext: omni.assets.plugins-0.0.0] startup [0.098s] [ext: omni.gpu_foundation-0.0.0] startup 2023-05-05 06:13:32 [96ms] [Warning] [carb] FrameworkImpl::setDefaultPlugin(client: omni.gpu_foundation_factory.plugin, desc : [carb::graphics::Graphics v2.11], plugin : carb.graphics-vulkan.plugin) failed. Plugin selection is locked, because the interface was previously acquired by: 2023-05-05 06:13:32 [95ms] [Warning] [carb] FrameworkImpl::setDefaultPlugin(client: omni.gpu_foundation_factory.plugin, desc : [carb::graphics::Graphics v2.11], plugin : carb.graphics-vulkan.plugin) failed. Plugin selection is locked, because the interface was previously acquired by: [0.106s] [ext: carb.windowing.plugins-1.0.0] startup [0.105s] [ext: carb.windowing.plugins-1.0.0] startup 2023-05-05 06:13:32 [105ms] [Warning] [carb.windowing-glfw.gamepad] Joystick with unknown remapping detected (will be ignored): ASRock LED Controller [03000000ce260000a201000010010000] [0.113s] [ext: omni.kit.renderer.init-0.0.0] startup 2023-05-05 06:13:32 [105ms] [Warning] [carb.windowing-glfw.gamepad] Joystick with unknown remapping detected (will be ignored): ASRock LED Controller [03000000ce260000a201000010010000] [0.112s] [ext: omni.kit.renderer.init-0.0.0] startup
2023-05-05 06:13:33 [873ms] [Warning] [gpu.foundation.plugin] IOMMU is enabled. Found 50 items in /sys/kernel/iommu_groups/. 2023-05-05 06:13:33 [875ms] [Warning] [gpu.foundation.plugin] IOMMU is enabled. Found 50 items in /sys/kernel/iommu_groups/. 2023-05-05 06:13:33 [873ms] [Warning] [gpu.foundation.plugin] On Linux only, CUDA and the display driver does not support IOMMU-enabled bare-metal PCIe peer to peer memory copy. However, CUDA and the display driver does support IOMMU via VM pass through. As a consequence, users on Linux, when running on a native bare metal system, should disable the IOMMU. The IOMMU should be enabled and the VFIO driver be used as a PCIe pass through for virtual machines. 2023-05-05 06:13:33 [875ms] [Warning] [gpu.foundation.plugin] On Linux only, CUDA and the display driver does not support IOMMU-enabled bare-metal PCIe peer to peer memory copy. However, CUDA and the display driver does support IOMMU via VM pass through. As a consequence, users on Linux, when running on a native bare metal system, should disable the IOMMU. The IOMMU should be enabled and the VFIO driver be used as a PCIe pass through for virtual machines. 2023-05-05 06:13:33 [875ms] [Warning] [gpu.foundation.plugin] 2023-05-05 06:13:33 [873ms] [Warning] [gpu.foundation.plugin] 2023-05-05 06:13:33 [875ms] [Warning] [gpu.foundation.plugin] ----------------------------------------------------------------------- 2023-05-05 06:13:33 [873ms] [Warning] [gpu.foundation.plugin] ----------------------------------------------------------------------- 2023-05-05 06:13:33 [875ms] [Warning] [gpu.foundation.plugin] !!!!! Local system validation failed! Incorrect configuration detected. 2023-05-05 06:13:33 [875ms] [Warning] [gpu.foundation.plugin] Summary below. Details above. 2023-05-05 06:13:33 [873ms] [Warning] [gpu.foundation.plugin] !!!!! Local system validation failed! Incorrect configuration detected. 2023-05-05 06:13:33 [875ms] [Warning] [gpu.foundation.plugin] ----------------------------------------------------------------------- 2023-05-05 06:13:33 [873ms] [Warning] [gpu.foundation.plugin] Summary below. Details above. 2023-05-05 06:13:33 [875ms] [Warning] [gpu.foundation.plugin] 2023-05-05 06:13:33 [873ms] [Warning] [gpu.foundation.plugin] ----------------------------------------------------------------------- 2023-05-05 06:13:33 [875ms] [Warning] [gpu.foundation.plugin] - ECC: OK 2023-05-05 06:13:33 [875ms] [Warning] [gpu.foundation.plugin] - IOMMU: FAILED 2023-05-05 06:13:33 [873ms] [Warning] [gpu.foundation.plugin] 2023-05-05 06:13:33 [875ms] [Warning] [gpu.foundation.plugin] 2023-05-05 06:13:33 [875ms] [Warning] [gpu.foundation.plugin] ----------------------------------------------------------------------- 2023-05-05 06:13:33 [873ms] [Warning] [gpu.foundation.plugin] - ECC: OK 2023-05-05 06:13:33 [873ms] [Warning] [gpu.foundation.plugin] - IOMMU: FAILED 2023-05-05 06:13:33 [873ms] [Warning] [gpu.foundation.plugin] 2023-05-05 06:13:33 [873ms] [Warning] [gpu.foundation.plugin] ----------------------------------------------------------------------- [1.630s] [ext: omni.kit.pipapi-0.0.0] startup [1.634s] [ext: omni.kit.pip_archive-0.0.0] startup [1.641s] [ext: omni.kit.loop-isaac-1.0.0] startup [1.641s] [ext: omni.kit.async_engine-0.0.0] startup [1.643s] [ext: omni.kit.test-0.0.0] startup [1.645s] [ext: omni.kit.pipapi-0.0.0] startup [1.649s] [ext: omni.kit.pip_archive-0.0.0] startup [1.656s] [ext: omni.kit.loop-isaac-1.0.0] startup [1.656s] [ext: omni.kit.async_engine-0.0.0] startup [1.658s] [ext: omni.kit.test-0.0.0] startup [1.664s] [ext: omni.usd.config-1.0.0] startup [1.669s] [ext: omni.usd.libs-1.0.0] startup [1.680s] [ext: omni.usd.config-1.0.0] startup [1.686s] [ext: omni.usd.libs-1.0.0] startup [1.832s] [ext: omni.isaac.core_archive-2.0.1] startup [1.840s] [ext: omni.pip.torch-1_13_0-0.1.4] startup [1.842s] [ext: omni.isaac.ml_archive-1.1.0] startup [1.843s] [ext: omni.client-0.1.1] startup [1.843s] [ext: omni.isaac.core_archive-2.0.1] startup [1.852s] [ext: omni.appwindow-1.0.1] startup [1.850s] [ext: omni.pip.torch-1_13_0-0.1.4] startup [1.854s] [ext: omni.kit.renderer.core-0.0.0] startup [1.853s] [ext: omni.isaac.ml_archive-1.1.0] startup [1.854s] [ext: omni.client-0.1.1] startup [1.858s] [ext: omni.kit.renderer.capture-0.0.0] startup [1.860s] [ext: omni.kit.renderer.imgui-0.0.0] startup [1.862s] [ext: omni.appwindow-1.0.1] startup [1.865s] [ext: omni.kit.renderer.core-0.0.0] startup [1.874s] [ext: omni.kit.renderer.capture-0.0.0] startup [1.877s] [ext: omni.kit.renderer.imgui-0.0.0] startup [1.912s] [ext: carb.audio-0.1.0] startup [1.929s] [ext: carb.audio-0.1.0] startup [1.932s] [ext: omni.ui-2.12.23] startup [1.939s] [ext: omni.ui-2.12.23] startup [1.942s] [ext: omni.uiaudio-1.0.0] startup [1.944s] [ext: omni.kit.mainwindow-1.0.0] startup [1.946s] [ext: omni.kit.uiapp-0.0.0] startup [1.946s] [ext: omni.usd.schema.physics-1.0.0] startup [1.949s] [ext: omni.uiaudio-1.0.0] startup [1.951s] [ext: omni.kit.mainwindow-1.0.0] startup [1.953s] [ext: omni.kit.uiapp-0.0.0] startup [1.953s] [ext: omni.usd.schema.physics-1.0.0] startup [1.991s] [ext: omni.usd.schema.audio-0.0.0] startup [1.999s] [ext: omni.usd.schema.audio-0.0.0] startup [2.001s] [ext: omni.usd.schema.semantics-0.0.0] startup [2.007s] [ext: omni.usd.schema.semantics-0.0.0] startup [2.012s] [ext: omni.usd.schema.omniscripting-1.0.0] startup [2.020s] [ext: omni.usd.schema.omniscripting-1.0.0] startup [2.021s] [ext: omni.usd.schema.geospatial-0.0.0] startup [2.029s] [ext: omni.usd.schema.geospatial-0.0.0] startup [2.032s] [ext: omni.usd.schema.physx-0.0.0] startup [2.042s] [ext: omni.usd.schema.physx-0.0.0] startup [2.061s] [ext: omni.usd.schema.anim-0.0.0] startup [2.074s] [ext: omni.usd.schema.anim-0.0.0] startup [2.142s] [ext: omni.usd.schema.omnigraph-1.0.0] startup [2.151s] [ext: omni.gpucompute.plugins-0.0.0] startup [2.152s] [ext: omni.hydra.scene_delegate-0.3.0] startup [2.155s] [ext: omni.usd.schema.omnigraph-1.0.0] startup [2.159s] [ext: omni.hydra.usdrt_delegate-4.1.1] startup [2.165s] [ext: omni.gpucompute.plugins-0.0.0] startup [2.165s] [ext: omni.hydra.scene_delegate-0.3.0] startup [2.172s] [ext: omni.hydra.usdrt_delegate-4.1.1] startup [2.175s] [ext: omni.usdphysics-104.1.6-5.1] startup [2.178s] [ext: omni.kit.window.popup_dialog-2.0.15] startup [2.185s] [ext: omni.kit.actions.core-1.0.0] startup [2.187s] [ext: omni.kit.audiodeviceenum-1.0.0] startup [2.189s] [ext: omni.usdphysics-104.1.6-5.1] startup [2.189s] [ext: omni.kit.widget.nucleus_connector-1.0.2] startup [2.192s] [ext: omni.kit.window.popup_dialog-2.0.15] startup [2.192s] [ext: omni.kit.commands-1.4.5] startup [2.197s] [ext: omni.kvdb-0.0.0] startup [2.199s] [ext: omni.kit.actions.core-1.0.0] startup [2.202s] [ext: omni.kit.audiodeviceenum-1.0.0] startup [2.200s] [ext: omni.convexdecomposition-104.1.6-5.1] startup [2.204s] [ext: omni.kit.widget.nucleus_connector-1.0.2] startup [2.204s] [ext: omni.usd.core-1.0.0] startup [2.207s] [ext: omni.kit.commands-1.4.5] startup [2.206s] [ext: omni.timeline-1.0.5] startup [2.209s] [ext: omni.kit.numpy.common-0.1.0] startup [2.212s] [ext: omni.kvdb-0.0.0] startup [2.211s] [ext: omni.usd-1.6.23] startup [2.215s] [ext: omni.convexdecomposition-104.1.6-5.1] startup [2.219s] [ext: omni.usd.core-1.0.0] startup [2.221s] [ext: omni.timeline-1.0.5] startup [2.224s] [ext: omni.kit.numpy.common-0.1.0] startup [2.226s] [ext: omni.usd-1.6.23] startup [2.256s] [ext: omni.localcache-0.0.0] startup [2.259s] [ext: omni.isaac.version-1.0.0] startup [2.260s] [ext: omni.physx-104.1.6-5.1] startup [2.271s] [ext: omni.localcache-0.0.0] startup [2.274s] [ext: omni.isaac.version-1.0.0] startup [2.275s] [ext: omni.physx-104.1.6-5.1] startup 2023-05-05 06:13:34 [2,282ms] [Warning] [omni.kvdb.plugin] wasn't able to load the meta database, trying to repair it ... [2.295s] [ext: omni.kit.widget.prompt-1.0.4] startup [2.296s] [ext: omni.isaac.dynamic_control-1.2.2] startup [2.302s] [ext: omni.kit.menu.utils-1.4.7] startup [2.373s] [ext: omni.kit.widget.path_field-2.0.4] startup [2.374s] [ext: omni.kit.notification_manager-1.0.4] startup [2.376s] [ext: omni.kit.search_core-1.0.1] startup [2.377s] [ext: omni.kit.widget.browser_bar-2.0.4] startup [2.378s] [ext: omni.kit.widget.versioning-1.3.8] startup [2.379s] [ext: omni.kit.widget.filebrowser-2.3.8] startup [2.394s] [ext: omni.kit.collaboration.channel_manager-1.0.9] startup [2.397s] [ext: omni.iray.libs-0.0.0] startup [2.400s] [ext: omni.kit.window.filepicker-2.7.8] startup [2.450s] [ext: omni.kit.usd.layers-2.0.10] startup [2.458s] [ext: omni.mdl.neuraylib-0.1.0] startup [2.461s] [ext: omni.kit.window.file_importer-1.0.8] startup [2.462s] [ext: omni.volume-0.1.0] startup [2.465s] [ext: omni.mdl-0.1.0] startup [2.482s] [ext: omni.kit.clipboard-1.0.0] startup [2.483s] [ext: omni.hydra.rtx-0.1.0] startup [2.489s] [ext: omni.kit.material.library-1.3.21] startup [2.494s] [ext: omni.activity.core-1.0.1] startup [2.500s] [ext: omni.kit.hydra_texture-1.0.11] startup [2.506s] [ext: omni.kit.menu.create-1.0.8] startup [2.507s] [ext: omni.kit.viewport.registry-104.0.2] startup [2.510s] [ext: omni.kit.widget.viewport-104.1.15] startup [2.511s] [ext: omni.kit.context_menu-1.5.10] startup [2.515s] [ext: omni.kit.window.file_exporter-1.0.10] startup [2.516s] [ext: omni.ui.scene-1.5.17] startup [2.522s] [ext: omni.kit.window.property-1.8.2] startup [2.523s] [ext: omni.kit.widget.stage-2.7.18] startup [2.526s] [ext: omni.kit.viewport.window-104.1.18] startup 2023-05-05 06:13:35 [2,520ms] [Warning] [omni.kit.ui] Failed to set value for menu with path: Window/Viewport/Viewport Next 1. It doesn't exist. [2.553s] [ext: omni.kit.property.usd-3.18.12] startup [2.561s] [ext: omni.kit.viewport.utility-1.0.12] startup [2.561s] [ext: omni.kit.widget.graph-1.5.3] startup [2.592s] [ext: omni.inspect-1.0.1] startup [2.595s] [ext: omni.kit.widget.text_editor-1.0.2] startup [2.596s] [ext: omni.ui_query-1.1.1] startup [2.599s] [ext: omni.kit.window.extensions-1.1.1] startup [2.602s] [ext: omni.graph.core-2.64.7] startup [2.605s] [ext: omni.kit.primitive.mesh-1.0.6] startup [2.609s] [ext: omni.kit.ui_test-1.2.9] startup [2.611s] [ext: omni.kit.widget.settings-1.0.1] startup [2.612s] [ext: omni.graph.tools-1.17.0] startup [2.626s] [ext: omni.kit.usd_undo-0.1.1] startup [2.628s] [ext: omni.kit.stage_templates-1.1.11] startup [2.630s] [ext: omni.kit.widget.searchfield-1.0.8] startup [2.631s] [ext: omni.kit.widget.zoombar-1.0.4] startup [2.631s] [ext: omni.kit.window.preferences-1.3.7] startup [2.672s] [ext: omni.graph-1.50.1] startup [2.785s] [ext: omni.debugdraw-0.1.1] startup [2.791s] [ext: omni.kit.browser.core-2.2.1] startup [2.796s] [ext: omni.graph.ui-1.24.1] startup [2.825s] [ext: omni.kit.window.file-1.3.30] startup [2.827s] [ext: omni.kit.browser.folder.core-1.7.1] startup [2.829s] [ext: omni.graph.nodes-1.48.3] startup [2.840s] [ext: omni.graph.action-1.31.0] startup [2.847s] [ext: omni.kit.browser.sample-1.2.4] startup [2.849s] [ext: omni.syntheticdata-0.2.4] startup 2023-05-05 06:13:35 [2,852ms] [Warning] [omni.syntheticdata.scripts.extension] SyntheticData extension needs at least a stageFrameHistoryCount of 3 [2.863s] [ext: omni.physics.tensors-0.1.0] startup [2.870s] [ext: omni.warp-0.6.1] startup Warp 0.6.1 initialized: CUDA Toolkit: 11.5, Driver: 12.1 Devices: "cpu" | x86_64 "cuda:0" | NVIDIA GeForce RTX 4090 (sm_89) "cuda:1" | NVIDIA GeForce RTX 4090 (sm_89) Kernel cache: /home/bizon/.cache/warp/0.6.1 2023-05-05 06:13:35 [2,874ms] [Warning] [omni.kvdb.plugin] repair failed [2.881s] [ext: omni.kit.widget.prompt-1.0.4] startup [2.882s] [ext: omni.isaac.dynamic_control-1.2.2] startup [2.889s] [ext: omni.kit.menu.utils-1.4.7] startup [2.960s] [ext: omni.kit.widget.path_field-2.0.4] startup [2.961s] [ext: omni.kit.notification_manager-1.0.4] startup [2.963s] [ext: omni.kit.search_core-1.0.1] startup [2.965s] [ext: omni.kit.widget.browser_bar-2.0.4] startup [2.965s] [ext: omni.kit.widget.versioning-1.3.8] startup [2.967s] [ext: omni.kit.widget.filebrowser-2.3.8] startup [2.981s] [ext: omni.kit.collaboration.channel_manager-1.0.9] startup [2.984s] [ext: omni.iray.libs-0.0.0] startup [2.986s] [ext: omni.kit.window.filepicker-2.7.8] startup [3.036s] [ext: omni.kit.usd.layers-2.0.10] startup [3.043s] [ext: omni.mdl.neuraylib-0.1.0] startup [3.044s] [ext: omni.graph.scriptnode-0.9.3] startup [3.047s] [ext: omni.kit.window.file_importer-1.0.8] startup [3.047s] [ext: omni.volume-0.1.0] startup [3.046s] [ext: omni.command.usd-1.0.1] startup [3.050s] [ext: omni.mdl-0.1.0] startup [3.049s] [ext: omni.physx.tensors-0.1.0] startup [3.055s] [ext: omni.replicator.core-1.6.4] startup [3.066s] [ext: omni.kit.clipboard-1.0.0] startup [3.067s] [ext: omni.hydra.rtx-0.1.0] startup [3.073s] [ext: omni.kit.material.library-1.3.21] startup [3.078s] [ext: omni.activity.core-1.0.1] startup 2023-05-05 06:13:35 [3,069ms] [Warning] [omni.replicator.core.scripts.annotators] Annotator PostProcessDispatch is already registered, overwriting annotator template [3.084s] [ext: omni.kit.hydra_texture-1.0.11] startup [3.090s] [ext: omni.kit.menu.create-1.0.8] startup [3.091s] [ext: omni.kit.viewport.registry-104.0.2] startup [3.093s] [ext: omni.kit.widget.viewport-104.1.15] startup [3.095s] [ext: omni.kit.context_menu-1.5.10] startup [3.099s] [ext: omni.kit.window.file_exporter-1.0.10] startup [3.100s] [ext: omni.ui.scene-1.5.17] startup [3.105s] [ext: omni.kit.window.property-1.8.2] startup [3.107s] [ext: omni.kit.widget.stage-2.7.18] startup [3.110s] [ext: omni.kit.viewport.window-104.1.18] startup 2023-05-05 06:13:35 [3,104ms] [Warning] [omni.kit.ui] Failed to set value for menu with path: Window/Viewport/Viewport Next 1. It doesn't exist. [3.136s] [ext: omni.kit.property.usd-3.18.12] startup [3.141s] [ext: omni.isaac.core-1.42.0] startup [3.144s] [ext: omni.kit.viewport.utility-1.0.12] startup [3.145s] [ext: omni.kit.widget.graph-1.5.3] startup [3.174s] [ext: omni.inspect-1.0.1] startup [3.177s] [ext: omni.kit.widget.text_editor-1.0.2] startup [3.178s] [ext: omni.ui_query-1.1.1] startup [3.181s] [ext: omni.kit.window.extensions-1.1.1] startup [3.184s] [ext: omni.graph.core-2.64.7] startup [3.188s] [ext: omni.kit.primitive.mesh-1.0.6] startup [3.191s] [ext: omni.kit.ui_test-1.2.9] startup [3.193s] [ext: omni.kit.widget.settings-1.0.1] startup [3.194s] [ext: omni.graph.tools-1.17.0] startup [3.208s] [ext: omni.kit.usd_undo-0.1.1] startup [3.209s] [ext: omni.kit.stage_templates-1.1.11] startup [3.211s] [ext: omni.kit.widget.searchfield-1.0.8] startup [3.212s] [ext: omni.kit.widget.zoombar-1.0.4] startup [3.213s] [ext: omni.kit.window.preferences-1.3.7] startup [3.214s] [ext: omni.graph.bundle.action-1.3.0] startup [3.214s] [ext: omni.replicator.isaac-1.7.3] startup [3.224s] [ext: omni.kit.window.title-1.1.2] startup [3.225s] [ext: omni.isaac.kit-1.2.1] startup [3.225s] [ext: omni.isaac.sim.python.gym.headless-2022.2.0] startup [3.226s] Simulation App Starting [3.253s] [ext: omni.graph-1.50.1] startup [3.368s] [ext: omni.debugdraw-0.1.1] startup [3.374s] [ext: omni.kit.browser.core-2.2.1] startup [3.379s] [ext: omni.graph.ui-1.24.1] startup [3.408s] [ext: omni.kit.window.file-1.3.30] startup [3.410s] [ext: omni.kit.browser.folder.core-1.7.1] startup [3.413s] [ext: omni.graph.nodes-1.48.3] startup [3.423s] [ext: omni.graph.action-1.31.0] startup [3.430s] [ext: omni.kit.browser.sample-1.2.4] startup [3.432s] [ext: omni.syntheticdata-0.2.4] startup 2023-05-05 06:13:36 [3,435ms] [Warning] [omni.syntheticdata.scripts.extension] SyntheticData extension needs at least a stageFrameHistoryCount of 3 [3.446s] [ext: omni.physics.tensors-0.1.0] startup [3.453s] [ext: omni.warp-0.6.1] startup Warp 0.6.1 initialized: CUDA Toolkit: 11.5, Driver: 12.1 Devices: "cpu" | x86_64 "cuda:0" | NVIDIA GeForce RTX 4090 (sm_89) "cuda:1" | NVIDIA GeForce RTX 4090 (sm_89) Kernel cache: /home/bizon/.cache/warp/0.6.1 [3.623s] [ext: omni.graph.scriptnode-0.9.3] startup [3.625s] [ext: omni.command.usd-1.0.1] startup [3.628s] [ext: omni.physx.tensors-0.1.0] startup [3.634s] [ext: omni.replicator.core-1.6.4] startup 2023-05-05 06:13:36 [3,648ms] [Warning] [omni.replicator.core.scripts.annotators] Annotator PostProcessDispatch is already registered, overwriting annotator template [3.717s] [ext: omni.isaac.core-1.42.0] startup [3.790s] [ext: omni.graph.bundle.action-1.3.0] startup [3.790s] [ext: omni.replicator.isaac-1.7.3] startup [3.800s] [ext: omni.kit.window.title-1.1.2] startup [3.801s] [ext: omni.isaac.kit-1.2.1] startup [3.801s] [ext: omni.isaac.sim.python.gym.headless-2022.2.0] startup [3.801s] Simulation App Starting [7.783s] app ready [7.792s] app ready [7.996s] Simulation App Startup Complete task: name: Ant physics_engine: physx env: numEnvs: 4096 envSpacing: 5 episodeLength: 1000 enableDebugVis: False clipActions: 1.0 powerScale: 0.5 controlFrequencyInv: 2 headingWeight: 0.5 upWeight: 0.1 actionsCost: 0.005 energyCost: 0.05 dofVelocityScale: 0.2 angularVelocityScale: 1.0 contactForceScale: 0.1 jointsAtLimitCost: 0.1 deathCost: -2.0 terminationHeight: 0.31 alive_reward_scale: 0.5 sim: dt: 0.0083 use_gpu_pipeline: True gravity: [0.0, 0.0, -9.81] add_ground_plane: True add_distant_light: True use_flatcache: True enable_scene_query_support: False disable_contact_processing: False enable_cameras: False default_physics_material: static_friction: 1.0 dynamic_friction: 1.0 restitution: 0.0 physx: worker_thread_count: 4 solver_type: 1 use_gpu: True solver_position_iteration_count: 4 solver_velocity_iteration_count: 0 contact_offset: 0.02 rest_offset: 0.0 bounce_threshold_velocity: 0.2 friction_offset_threshold: 0.04 friction_correlation_distance: 0.025 enable_sleeping: True enable_stabilization: True max_depenetration_velocity: 10.0 gpu_max_rigid_contact_count: 524288 gpu_max_rigid_patch_count: 81920 gpu_found_lost_pairs_capacity: 8192 gpu_found_lost_aggregate_pairs_capacity: 262144 gpu_total_aggregate_pairs_capacity: 8192 gpu_max_soft_body_contacts: 1048576 gpu_max_particle_contacts: 1048576 gpu_heap_capacity: 67108864 gpu_temp_buffer_capacity: 16777216 gpu_max_num_partitions: 8 Ant: override_usd_defaults: False enable_self_collisions: False enable_gyroscopic_forces: True solver_position_iteration_count: 4 solver_velocity_iteration_count: 0 sleep_threshold: 0.005 stabilization_threshold: 0.001 density: -1 max_depenetration_velocity: 10.0 train: params: seed: 42 algo: name: a2c_continuous model: name: continuous_a2c_logstd network: name: actor_critic separate: False space: continuous: mu_activation: None sigma_activation: None mu_init: name: default sigma_init: name: const_initializer val: 0 fixed_sigma: True mlp: units: [256, 128, 64] activation: elu d2rl: False initializer: name: default regularizer: name: None load_checkpoint: False load_path: config: name: Ant full_experiment_name: Ant env_name: rlgpu device: cuda:1 device_name: cuda:1 multi_gpu: True ppo: True mixed_precision: True normalize_input: True normalize_value: True value_bootstrap: True num_actors: 4096 reward_shaper: scale_value: 0.01 normalize_advantage: True gamma: 0.99 tau: 0.95 learning_rate: 0.0003 lr_schedule: adaptive schedule_type: legacy kl_threshold: 0.008 score_to_win: 20000 max_epochs: 500 save_best_after: 100 save_frequency: 50 grad_norm: 1.0 entropy_coef: 0.0 truncate_grads: True e_clip: 0.2 horizon_length: 16 minibatch_size: 32768 mini_epochs: 4 critic_coef: 2 clip_value: True seq_len: 4 bounds_loss_coef: 0.0001 task_name: Ant experiment: num_envs: seed: 42 torch_deterministic: False max_iterations: physics_engine: physx pipeline: gpu sim_device: gpu device_id: 1 rl_device: cuda:1 multi_gpu: True num_threads: 4 solver_type: 1 test: False checkpoint: headless: True enable_livestream: False mt_timeout: 30 wandb_activate: False wandb_group: wandb_name: Ant wandb_entity: wandb_project: omniisaacgymenvs Setting seed: 42 Sim params does not have attribute: physx Sim params does not have attribute: Ant Pipeline: GPU Pipeline Device: cuda:1 Sim Device: GPU [8.015s] Simulation App Startup Complete task: name: Ant physics_engine: physx env: numEnvs: 4096 envSpacing: 5 episodeLength: 1000 enableDebugVis: False clipActions: 1.0 powerScale: 0.5 controlFrequencyInv: 2 headingWeight: 0.5 upWeight: 0.1 actionsCost: 0.005 energyCost: 0.05 dofVelocityScale: 0.2 angularVelocityScale: 1.0 contactForceScale: 0.1 jointsAtLimitCost: 0.1 deathCost: -2.0 terminationHeight: 0.31 alive_reward_scale: 0.5 sim: dt: 0.0083 use_gpu_pipeline: True gravity: [0.0, 0.0, -9.81] add_ground_plane: True add_distant_light: True use_flatcache: True enable_scene_query_support: False disable_contact_processing: False enable_cameras: False default_physics_material: static_friction: 1.0 dynamic_friction: 1.0 restitution: 0.0 physx: worker_thread_count: 4 solver_type: 1 use_gpu: True solver_position_iteration_count: 4 solver_velocity_iteration_count: 0 contact_offset: 0.02 rest_offset: 0.0 bounce_threshold_velocity: 0.2 friction_offset_threshold: 0.04 friction_correlation_distance: 0.025 enable_sleeping: True enable_stabilization: True max_depenetration_velocity: 10.0 gpu_max_rigid_contact_count: 524288 gpu_max_rigid_patch_count: 81920 gpu_found_lost_pairs_capacity: 8192 gpu_found_lost_aggregate_pairs_capacity: 262144 gpu_total_aggregate_pairs_capacity: 8192 gpu_max_soft_body_contacts: 1048576 gpu_max_particle_contacts: 1048576 gpu_heap_capacity: 67108864 gpu_temp_buffer_capacity: 16777216 gpu_max_num_partitions: 8 Ant: override_usd_defaults: False enable_self_collisions: False enable_gyroscopic_forces: True solver_position_iteration_count: 4 solver_velocity_iteration_count: 0 sleep_threshold: 0.005 stabilization_threshold: 0.001 density: -1 max_depenetration_velocity: 10.0 train: params: seed: 42 algo: name: a2c_continuous model: name: continuous_a2c_logstd network: name: actor_critic separate: False space: continuous: mu_activation: None sigma_activation: None mu_init: name: default sigma_init: name: const_initializer val: 0 fixed_sigma: True mlp: units: [256, 128, 64] activation: elu d2rl: False initializer: name: default regularizer: name: None load_checkpoint: False load_path: config: name: Ant full_experiment_name: Ant env_name: rlgpu device: cuda:0 device_name: cuda:0 multi_gpu: True ppo: True mixed_precision: True normalize_input: True normalize_value: True value_bootstrap: True num_actors: 4096 reward_shaper: scale_value: 0.01 normalize_advantage: True gamma: 0.99 tau: 0.95 learning_rate: 0.0003 lr_schedule: adaptive schedule_type: legacy kl_threshold: 0.008 score_to_win: 20000 max_epochs: 500 save_best_after: 100 save_frequency: 50 grad_norm: 1.0 entropy_coef: 0.0 truncate_grads: True e_clip: 0.2 horizon_length: 16 minibatch_size: 32768 mini_epochs: 4 critic_coef: 2 clip_value: True seq_len: 4 bounds_loss_coef: 0.0001 task_name: Ant experiment: num_envs: seed: 42 torch_deterministic: False max_iterations: physics_engine: physx pipeline: gpu sim_device: gpu device_id: 0 rl_device: cuda:0 multi_gpu: True num_threads: 4 solver_type: 1 test: False checkpoint: headless: True enable_livestream: False mt_timeout: 30 wandb_activate: False wandb_group: wandb_name: Ant wandb_entity: wandb_project: omniisaacgymenvs Setting seed: 42 Sim params does not have attribute: physx Sim params does not have attribute: Ant Pipeline: GPU Pipeline Device: cuda:0 Sim Device: GPU Task Device: cuda:1 RL device: cuda:1 /home/bizon/anaconda3/envs/isaac-sim/lib/python3.7/site-packages/gym/spaces/box.py:84: UserWarning: WARN: Box bound precision lowered by casting to float32 logger.warn(f"Box bound precision lowered by casting to {self.dtype}") Task Device: cuda:0 RL device: cuda:0 /home/bizon/anaconda3/envs/isaac-sim/lib/python3.7/site-packages/gym/spaces/box.py:84: UserWarning: WARN: Box bound precision lowered by casting to float32 logger.warn(f"Box bound precision lowered by casting to {self.dtype}") [8.827s] [ext: omni.isaac.sim.python.gym.headless-2022.2.0] shutdown [8.827s] [ext: omni.replicator.isaac-1.7.3] shutdown [8.827s] [ext: omni.replicator.core-1.6.4] shutdown 2023-05-05 06:13:41 [8,820ms] [Warning] [omni.graph.core.plugin] Could not find category 'Replicator:Annotators' for removal 2023-05-05 06:13:41 [8,820ms] [Warning] [omni.graph.core.plugin] Could not find category 'Replicator:Annotators' for removal 2023-05-05 06:13:41 [8,820ms] [Warning] [omni.graph.core.plugin] Could not find category 'Replicator:Annotators' for removal 2023-05-05 06:13:41 [8,820ms] [Warning] [omni.graph.core.plugin] Could not find category 'Replicator:Annotators' for removal 2023-05-05 06:13:41 [8,820ms] [Warning] [omni.graph.core.plugin] Could not find category 'Replicator:Annotators' for removal 2023-05-05 06:13:41 [8,820ms] [Warning] [omni.graph.core.plugin] Could not find category 'Replicator:Annotators' for removal 2023-05-05 06:13:41 [8,820ms] [Warning] [omni.graph.core.plugin] Could not find category 'Replicator:Annotators' for removal 2023-05-05 06:13:41 [8,820ms] [Warning] [omni.graph.core.plugin] Could not find category 'Replicator:Annotators' for removal 2023-05-05 06:13:41 [8,821ms] [Warning] [carb] [Plugin: omni.replicator.core.plugin] Module /home/bizon/.local/share/ov/pkg/isaac_sim-2022.2.0/extscache/omni.replicator.core-1.6.4+104.1.lx64.r.cp37/bin/libomni.replicator.core.plugin.so remained loaded after unload request [8.833s] [ext: omni.syntheticdata-0.2.4] shutdown [8.834s] [ext: omni.isaac.core-1.42.0] shutdown [8.834s] [ext: omni.graph.scriptnode-0.9.3] shutdown [8.834s] [ext: omni.warp-0.6.1] shutdown [8.838s] [ext: omni.graph.bundle.action-1.3.0] shutdown [8.838s] [ext: omni.graph.nodes-1.48.3] shutdown [8.839s] [ext: omni.graph.action-1.31.0] shutdown [8.839s] [ext: omni.graph.ui-1.24.1] shutdown 2023-05-05 06:13:41 [8,834ms] [Warning] [omni.kit.ui] Failed to remove menu with path: Window/Visual Scripting/Toolkit. It doesn't exist. 2023-05-05 06:13:41 [8,834ms] [Warning] [omni.kit.menu.utils.scripts.utils] omni.kit.menu.utils remove_menu_items "Create" failed list.remove(x): x not in list 2023-05-05 06:13:41 [8,834ms] [Warning] [omni.kit.ui] Failed to remove menu with path: Window/Visual Scripting/Toolkit. It doesn't exist. 2023-05-05 06:13:41 [8,834ms] [Warning] [omni.kit.ui] Failed to remove menu with path: Window/Visual Scripting/Node Description Editor. It doesn't exist. [9.077s] [ext: omni.kit.viewport.utility-1.0.12] shutdown [9.077s] [ext: omni.kit.browser.sample-1.2.4] shutdown [9.078s] [ext: omni.kit.browser.folder.core-1.7.1] shutdown [9.079s] [ext: omni.kit.browser.core-2.2.1] shutdown [9.079s] [ext: omni.kit.viewport.window-104.1.18] shutdown 2023-05-05 06:13:41 [9,094ms] [Warning] [omni.ext._impl._internal] omni.kit.viewport.window-104.1.18 -> <class 'omni.kit.viewport.window.extension.ViewportWindowExtension'>: extension object is still alive, something holds a reference on it. References: ["[0]:type: <class 'frame'>, id: 334733824", "[1]:type: <class 'frame'>, id: 334582032", "[2]:type: <class 'frame'>, id: 111020256"] [9.448s] [ext: omni.isaac.sim.python.gym.headless-2022.2.0] shutdown [9.448s] [ext: omni.replicator.isaac-1.7.3] shutdown [9.448s] [ext: omni.replicator.core-1.6.4] shutdown 2023-05-05 06:13:42 [9,441ms] [Warning] [omni.graph.core.plugin] Could not find category 'Replicator:Annotators' for removal 2023-05-05 06:13:42 [9,441ms] [Warning] [omni.graph.core.plugin] Could not find category 'Replicator:Annotators' for removal 2023-05-05 06:13:42 [9,441ms] [Warning] [omni.graph.core.plugin] Could not find category 'Replicator:Annotators' for removal 2023-05-05 06:13:42 [9,441ms] [Warning] [omni.graph.core.plugin] Could not find category 'Replicator:Annotators' for removal 2023-05-05 06:13:42 [9,441ms] [Warning] [omni.graph.core.plugin] Could not find category 'Replicator:Annotators' for removal 2023-05-05 06:13:42 [9,441ms] [Warning] [omni.graph.core.plugin] Could not find category 'Replicator:Annotators' for removal 2023-05-05 06:13:42 [9,441ms] [Warning] [omni.graph.core.plugin] Could not find category 'Replicator:Annotators' for removal 2023-05-05 06:13:42 [9,441ms] [Warning] [omni.graph.core.plugin] Could not find category 'Replicator:Annotators' for removal 2023-05-05 06:13:42 [9,441ms] [Warning] [carb] [Plugin: omni.replicator.core.plugin] Module /home/bizon/.local/share/ov/pkg/isaac_sim-2022.2.0/extscache/omni.replicator.core-1.6.4+104.1.lx64.r.cp37/bin/libomni.replicator.core.plugin.so remained loaded after unload request [9.454s] [ext: omni.warp-0.6.1] shutdown [9.458s] [ext: omni.kit.browser.sample-1.2.4] shutdown [9.459s] [ext: omni.kit.browser.folder.core-1.7.1] shutdown [9.459s] [ext: omni.kit.browser.core-2.2.1] shutdown [9.459s] [ext: omni.syntheticdata-0.2.4] shutdown [9.460s] [ext: omni.isaac.core-1.42.0] shutdown [9.460s] [ext: omni.graph.scriptnode-0.9.3] shutdown [9.460s] [ext: omni.graph.bundle.action-1.3.0] shutdown [9.460s] [ext: omni.graph.nodes-1.48.3] shutdown [9.461s] [ext: omni.graph.action-1.31.0] shutdown [9.461s] [ext: omni.graph.ui-1.24.1] shutdown 2023-05-05 06:13:42 [9,456ms] [Warning] [omni.kit.ui] Failed to remove menu with path: Window/Visual Scripting/Toolkit. It doesn't exist. 2023-05-05 06:13:42 [9,456ms] [Warning] [omni.kit.menu.utils.scripts.utils] omni.kit.menu.utils remove_menu_items "Create" failed list.remove(x): x not in list 2023-05-05 06:13:42 [9,456ms] [Warning] [omni.kit.ui] Failed to remove menu with path: Window/Visual Scripting/Toolkit. It doesn't exist. 2023-05-05 06:13:42 [9,456ms] [Warning] [omni.kit.ui] Failed to remove menu with path: Window/Visual Scripting/Node Description Editor. It doesn't exist. [9.697s] [ext: omni.kit.viewport.utility-1.0.12] shutdown [9.697s] [ext: omni.kit.viewport.window-104.1.18] shutdown 2023-05-05 06:13:42 [9,713ms] [Warning] [omni.ext._impl._internal] omni.kit.viewport.window-104.1.18 -> <class 'omni.kit.viewport.window.extension.ViewportWindowExtension'>: extension object is still alive, something holds a reference on it. References: ["[0]:type: <class 'frame'>, id: 343270400", "[1]:type: <class 'frame'>, id: 343118608", "[2]:type: <class 'frame'>, id: 119418496"] Error executing job with overrides: ['headless=True', 'task=Ant', 'multi_gpu=True'] Traceback (most recent call last): File "scripts/rlgames_train.py", line 115, in parse_hydra_configs task = initialize_task(cfg_dict, env) File "/home/bizon/eric/OmniIsaacGymEnvs/omniisaacgymenvs/utils/task_util.py", line 72, in initialize_task env.set_task(task=task, sim_params=sim_config.get_physics_params(), backend="torch", init_sim=init_sim) File "/home/bizon/eric/OmniIsaacGymEnvs/omniisaacgymenvs/envs/vec_env_rlgames.py", line 51, in set_task super().set_task(task, backend, sim_params, init_sim) File "/home/bizon/.local/share/ov/pkg/isaac_sim-2022.2.0/exts/omni.isaac.gym/omni/isaac/gym/vec_env/vec_env_base.py", line 80, in set_task self._world.reset() File "/home/bizon/.local/share/ov/pkg/isaac_sim-2022.2.0/exts/omni.isaac.core/omni/isaac/core/world/world.py", line 285, in reset task.post_reset() File "/home/bizon/eric/OmniIsaacGymEnvs/omniisaacgymenvs/tasks/ant.py", line 86, in post_reset LocomotionTask.post_reset(self) File "/home/bizon/eric/OmniIsaacGymEnvs/omniisaacgymenvs/tasks/shared/locomotion.py", line 174, in post_reset self.reset_idx(indices) File "/home/bizon/eric/OmniIsaacGymEnvs/omniisaacgymenvs/tasks/shared/locomotion.py", line 125, in reset_idx self.initial_dof_pos[env_ids] + dof_pos, self.dof_limits_lower, self.dof_limits_upper RuntimeError: indices should be either on cpu or on the same device as the indexed tensor (cuda:0)
Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace. self.seed = 42 Started to train Exact experiment name requested from command line: Ant WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 321900 closing signal SIGTERM WARNING:torch.distributed.elastic.multiprocessing.api:Unable to shutdown process 321900 via 15, forcefully exitting via 9 ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: -11) local_rank: 1 (pid: 321901) of binary: /home/bizon/anaconda3/envs/isaac-sim/bin/python Traceback (most recent call last): File "/home/bizon/anaconda3/envs/isaac-sim/lib/python3.7/runpy.py", line 193, in _run_module_as_main "main", mod_spec) File "/home/bizon/anaconda3/envs/isaac-sim/lib/python3.7/runpy.py", line 85, in _run_code exec(code, run_globals) File "/home/bizon/.local/share/ov/pkg/isaac_sim-2022.2.0/extscache/omni.pip.torch-1_13_0-0.1.4+104.1.lx64/torch-1-13-0/torch/distributed/run.py", line 766, in
main()
File "/home/bizon/.local/share/ov/pkg/isaac_sim-2022.2.0/extscache/omni.pip.torch-1_13_0-0.1.4+104.1.lx64/torch-1-13-0/torch/distributed/elastic/multiprocessing/errors/init.py", line 346, in wrapper
return f(*args, *kwargs)
File "/home/bizon/.local/share/ov/pkg/isaac_sim-2022.2.0/extscache/omni.pip.torch-1_13_0-0.1.4+104.1.lx64/torch-1-13-0/torch/distributed/run.py", line 762, in main
run(args)
File "/home/bizon/.local/share/ov/pkg/isaac_sim-2022.2.0/extscache/omni.pip.torch-1_13_0-0.1.4+104.1.lx64/torch-1-13-0/torch/distributed/run.py", line 756, in run
)(cmd_args)
File "/home/bizon/.local/share/ov/pkg/isaac_sim-2022.2.0/extscache/omni.pip.torch-1_13_0-0.1.4+104.1.lx64/torch-1-13-0/torch/distributed/launcher/api.py", line 132, in call
return launch_agent(self._config, self._entrypoint, list(args))
File "/home/bizon/.local/share/ov/pkg/isaac_sim-2022.2.0/extscache/omni.pip.torch-1_13_0-0.1.4+104.1.lx64/torch-1-13-0/torch/distributed/launcher/api.py", line 248, in launch_agent
failures=result.failures,
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
scripts/rlgames_train.py FAILED
Failures: