search

isaac-sim / OmniIsaacGymEnvs

Reinforcement Learning Environments for Omniverse Isaac Gym
Other
860 stars 218 forks source link

Examples segfault #20

Closed dHonerkamp closed 1 year ago

dHonerkamp commented 1 year ago

Hi,

I'm trying to run some of the examples. While the Ant task works fine, other robots lead to a segfault during the initialize_task() function. This includes both the cartpole and franke_cabinet tasks.

I am running this in the latest Isaac docker image nvcr.io/nvidia/isaac-sim:2022.1.1 on an RTX A6000 with 515.65.1 drivers

The error is

[Warning] [omni.isaac.kit.simulation_app] Modules: ['omniisaacgymenvs', 'omniisaacgymenvs.utils', 'omniisaacgymenvs.utils.hydra_cfg', 'omniisaacgymenvs.utils.hydra_cfg.hydra_utils', 'omniisaacgymenvs.utils.hydra_cfg.reformat', 'omniisaacgymenvs.utils.rlgames', 'omniisaacgymenvs.utils.rlgames.rlgames_utils', 'omniisaacgymenvs.utils.config_utils', 'omniisaacgymenvs.utils.config_utils.path_utils', 'omniisaacgymenvs.envs', 'omniisaacgymenvs.envs.vec_env_rlgames', 'omni.isaac.gym', 'omni.isaac.gym.vec_env', 'omni.isaac.gym.vec_env.vec_env_base', 'omni.isaac.gym.vec_env.vec_env_mt'] were loaded before SimulationApp was started and might not be loaded correctly.
[Warning] [omni.isaac.kit.simulation_app] Please check to make sure no extra omniverse or pxr modules are imported before the call to SimulationApp(...)
Starting kit application with the fillowing args:  ['/isaac-sim/exts/omni.isaac.kit/omni/isaac/kit/simulation_app.py', '/isaac-sim/apps/omni.isaac.sim.python.kit', '--/app/tokens/exe-path=/isaac-sim/kit', '--/persistent/app/viewport/displayOptions=3094', '--/rtx/materialDb/syncLoads=True', '--/rtx/hydra/materialSyncLoads=True--/omni.kit.plugin/syncUsdLoads=True', '--/app/renderer/resolution/width=1280', '--/app/renderer/resolution/height=720', '--/app/window/width=1440', '--/app/window/height=900', '--/renderer/multiGpu/enabled=True', '--ext-folder', '/isaac-sim/exts', '--ext-folder', '/isaac-sim/apps', '--/physics/cudaDevice=0', '--portable', '--allow-root']
Passing the following args to the base kit application:  ['--enable', 'omni.kit.livestream.native', '--no-window']
[Warning] [omni.kit.app.plugin] No crash reporter present, dumps uploading isn't available.
[Info] [carb] Logging to file: /isaac-sim/kit/logs/Kit/Isaac-Sim/2022.1/kit_20221209_085428.log
2022-12-09 08:54:28 [1ms] [Warning] [omni.ext.plugin] [ext: omni.drivesim.sensors.nv.lidar] Extensions config 'extension.toml' doesn't exist '/isaac-sim/exts/omni.drivesim.sensors.nv.lidar' or '/isaac-sim/exts/omni.drivesim.sensors.nv.lidar/config'
[0.115s] [ext: omni.stats-0.0.0] startup
[0.155s] [ext: omni.gpu_foundation-0.0.0] startup
2022-12-09 08:54:28 [153ms] [Warning] [carb] FrameworkImpl::setDefaultPlugin(client: omni.gpu_foundation_factory.plugin, desc : [carb::graphics::Graphics v2.5], plugin : carb.graphics-vulkan.plugin) failed. Plugin selection is locked, because the interface was previously acquired by: 
[0.166s] [ext: carb.windowing.plugins-1.0.0] startup
2022-12-09 08:54:28 [159ms] [Error] [carb.windowing-glfw.plugin] GLFW initialization failed.
2022-12-09 08:54:28 [159ms] [Error] [carb] Failed to startup plugin carb.windowing-glfw.plugin (interfaces: [carb::windowing::IGLContext v1.0],[carb::windowing::IWindowing v1.3]) (impl: carb.windowing-glfw.plugin)
[0.167s] [ext: omni.assets.plugins-0.0.0] startup
[0.168s] [ext: omni.kit.renderer.init-0.0.0] startup

|---------------------------------------------------------------------------------------------|
| Driver Version: 515.65.1      | Graphics API: Vulkan
|=============================================================================================|
| GPU | Name                             | Active | LDA | GPU Memory | Vendor-ID | LUID       |
|     |                                  |        |     |            | Device-ID | UUID       |
|---------------------------------------------------------------------------------------------|
| 0   | NVIDIA RTX A6000                 | Yes: 0 |     | 49386   MB | 10de      | 0          |
|     |                                  |        |     |            | 2230      | 31d81b8f.. |
|=============================================================================================|
| OS: Linux rlsim2, Version: 5.4.0-132-generic
| Processor: AMD Ryzen Threadripper PRO 3975WX 32-Cores      | Cores: Unknown | Logical: 64
|---------------------------------------------------------------------------------------------|
| Total Memory (MB): 128729 | Free Memory: 119711
| Total Page/Swap (MB): 8191 | Free Page/Swap: 8191
|---------------------------------------------------------------------------------------------|
2022-12-09 08:55:02 [34,347ms] [Error] [carb.windowing-glfw.plugin] GLFW initialization failed.
2022-12-09 08:55:02 [34,347ms] [Error] [carb] Failed to startup plugin carb.windowing-glfw.plugin (interfaces: [carb::windowing::IGLContext v1.0],[carb::windowing::IWindowing v1.3]) (impl: carb.windowing-glfw.plugin)
2022-12-09 08:55:02 [34,348ms] [Error] [carb.glinterop.plugin] GLInteropContext::init: carb::windowing is not available
2022-12-09 08:55:02 [34,348ms] [Warning] [gpu.foundation.plugin] Realm: no OpenGL interop context.
2022-12-09 08:55:02 [34,439ms] [Warning] [carb.cudainterop.plugin] On Linux only, CUDA and the display driver does not support IOMMU-enabled bare-metal PCIe peer to peer memory copy.
However, CUDA and the display driver does support IOMMU via VM pass through. As a consequence, users on Linux,
when running on a native bare metal system, should disable the IOMMU. The IOMMU should be enabled and the VFIO driver
be used as a PCIe pass through for virtual machines.
[34.456s] [ext: omni.kit.pipapi-0.0.0] startup
[34.477s] [ext: omni.kit.pip_archive-0.0.0] startup
[34.489s] [ext: omni.isaac.core_archive-1.2.0] startup
[34.533s] [ext: omni.usd.config-1.0.0] startup
[34.539s] [ext: omni.usd.libs-1.0.0] startup
[34.645s] [ext: omni.kit.pip_torch-1_11_0-0.1.3] startup
[34.713s] [ext: omni.isaac.ml_archive-1.1.0] startup
[34.720s] [ext: omni.kit.loop-isaac-0.1.0] startup
[34.721s] [ext: omni.kit.async_engine-0.0.0] startup
[34.725s] [ext: omni.appwindow-1.0.0] startup
2022-12-09 08:55:02 [34,718ms] [Error] [carb.windowing-glfw.plugin] GLFW initialization failed.
2022-12-09 08:55:02 [34,718ms] [Error] [carb] Failed to startup plugin carb.windowing-glfw.plugin (interfaces: [carb::windowing::IGLContext v1.0],[carb::windowing::IWindowing v1.3]) (impl: carb.windowing-glfw.plugin)
[34.731s] [ext: omni.client-0.1.0] startup
[34.744s] [ext: omni.kit.test-0.0.0] startup
[34.747s] [ext: omni.kit.renderer.core-0.0.0] startup
2022-12-09 08:55:02 [34,741ms] [Error] [carb.windowing-glfw.plugin] GLFW initialization failed.
2022-12-09 08:55:02 [34,741ms] [Error] [carb] Failed to startup plugin carb.windowing-glfw.plugin (interfaces: [carb::windowing::IGLContext v1.0],[carb::windowing::IWindowing v1.3]) (impl: carb.windowing-glfw.plugin)
2022-12-09 08:55:02 [34,742ms] [Error] [carb.windowing-glfw.plugin] GLFW initialization failed.
2022-12-09 08:55:02 [34,742ms] [Error] [carb] Failed to startup plugin carb.windowing-glfw.plugin (interfaces: [carb::windowing::IGLContext v1.0],[carb::windowing::IWindowing v1.3]) (impl: carb.windowing-glfw.plugin)
2022-12-09 08:55:02 [34,742ms] [Error] [carb.windowing-glfw.plugin] GLFW initialization failed.
2022-12-09 08:55:02 [34,742ms] [Error] [carb] Failed to startup plugin carb.windowing-glfw.plugin (interfaces: [carb::windowing::IGLContext v1.0],[carb::windowing::IWindowing v1.3]) (impl: carb.windowing-glfw.plugin)
2022-12-09 08:55:02 [34,743ms] [Error] [carb.windowing-glfw.plugin] GLFW initialization failed.
2022-12-09 08:55:02 [34,743ms] [Error] [carb] Failed to startup plugin carb.windowing-glfw.plugin (interfaces: [carb::windowing::IGLContext v1.0],[carb::windowing::IWindowing v1.3]) (impl: carb.windowing-glfw.plugin)
[34.841s] [ext: omni.ui-2.10.3] startup
[34.864s] [ext: carb.audio-0.1.0] startup
[34.868s] [ext: omni.kit.mainwindow-0.0.0] startup
[34.872s] [ext: omni.uiaudio-1.0.0] startup
[34.875s] [ext: omni.kit.uiapp-0.0.0] startup
[34.875s] [ext: omni.usd.schema.physics-1.0.0] startup
[34.935s] [ext: omni.usd.schema.isaac-0.2.0] startup
[35.029s] [ext: omni.usd.schema.omnigraph-1.0.0] startup
[35.049s] [ext: omni.usd.schema.physx-0.0.0] startup
[35.150s] [ext: omni.usd.schema.anim-0.0.0] startup
[35.264s] [ext: omni.usd.schema.audio-0.0.0] startup
[35.281s] [ext: omni.usd.schema.semantics-0.0.0] startup
[35.299s] [ext: omni.usd.schema.forcefield-0.0.0] startup
[35.318s] [ext: omni.timeline-1.0.2] startup
[35.323s] [ext: omni.hydra.scene_delegate-0.2.0] startup
[35.329s] [ext: omni.kit.commands-1.2.2] startup
[35.342s] [ext: omni.kit.audiodeviceenum-1.0.0] startup
[35.344s] [ext: omni.usd-1.5.3] startup
[35.396s] [ext: omni.kit.asset_converter-1.2.31] startup
[35.418s] [ext: omni.kit.widget.path_field-2.0.3] startup
[35.427s] [ext: omni.kit.search_core-1.0.2] startup
[35.430s] [ext: omni.kit.window.popup_dialog-2.0.8] startup
[35.440s] [ext: omni.kit.widget.browser_bar-2.0.3] startup
[35.443s] [ext: omni.kit.widget.filebrowser-2.2.27] startup
[35.452s] [ext: omni.mdl.neuraylib-0.1.0] startup
[35.460s] [ext: omni.kit.menu.utils-1.2.11] startup
[35.476s] [ext: omni.kit.notification_manager-1.0.5] startup
[35.481s] [ext: omni.kit.widget.versioning-1.3.8] startup
[35.485s] [ext: omni.kit.menu.create-1.0.2] startup
[35.486s] [ext: omni.kit.window.filepicker-2.4.30] startup
OmniAssetFileFormat
[35.573s] [ext: omni.mdl-0.1.0] startup
[35.601s] [ext: omni.kit.stage_templates-1.1.2] startup
[35.606s] [ext: omni.kit.window.file_exporter-1.0.4] startup
[35.608s] [ext: omni.kit.material.library-1.3.10] startup
[35.614s] [ext: omni.kit.window.file_importer-1.0.4] startup
[35.615s] [ext: omni.kit.window.drop_support-1.0.0] startup
[35.617s] [ext: omni.kit.window.file-1.3.16] startup
[35.620s] [ext: omni.kit.context_menu-1.3.9] startup
[35.627s] [ext: omni.kit.window.property-1.6.3] startup
[35.629s] [ext: omni.kit.window.content_browser-2.4.28] startup
[35.652s] [ext: omni.kit.widget.stage-2.6.15] startup
[35.659s] [ext: omni.renderer-rtx-0.0.0] startup
[35.659s] [ext: omni.kit.property.usd-3.14.9] startup
[35.731s] [ext: omni.kit.widget.settings-1.0.0] startup
[35.736s] [ext: omni.kit.widget.prompt-1.0.1] startup
[35.737s] [ext: omni.kit.widget.graph-1.4.3] startup
[35.754s] [ext: omni.kit.window.preferences-1.2.1] startup
[35.819s] [ext: omni.hydra.engine.stats-1.0.0] startup
[35.829s] [ext: omni.hydra.rtx-0.1.0] startup
[35.837s] [ext: omni.kit.viewport.legacy_gizmos-1.0.0] startup
[35.839s] [ext: omni.graph.core-2.29.1] startup
[35.842s] [ext: omni.graph.tools-1.4.0] startup
[35.952s] [ext: omni.kit.window.viewport-0.0.0] startup
[39.336s] [ext: omni.graph-1.23.0] startup
[39.439s] [ext: omni.debugdraw-0.1.0] startup
[39.448s] [ext: omni.ui_query-1.1.1] startup
[39.454s] [ext: omni.graph.ui-1.6.1] startup
[39.488s] [ext: omni.kit.ui_test-1.2.2] startup
[39.493s] [ext: omni.kvdb-0.0.0] startup
[39.500s] [ext: omni.graph.action-1.18.0] startup
[39.517s] [ext: omni.kit.widget.searchfield-1.0.6] startup
[39.520s] [ext: omni.convexdecomposition-1.4.15] startup
[39.527s] [ext: omni.localcache-0.0.0] startup
[39.535s] [ext: omni.usdphysics-1.4.15] startup
[39.537s] [ext: omni.graph.scriptnode-0.5.0] startup
[39.541s] [ext: omni.physx-1.4.15-5.1] startup
[39.575s] [ext: omni.kit.numpy.common-0.1.0] startup
[39.584s] [ext: omni.physics.tensors-0.1.0] startup
[39.596s] [ext: omni.graph.nodes-1.26.0] startup
[39.615s] [ext: omni.isaac.dynamic_control-1.1.0] startup
[39.625s] [ext: omni.isaac.version-1.0.0] startup
[39.627s] [ext: omni.physx.tensors-0.1.0] startup
[39.631s] [ext: omni.syntheticdata-0.2.1] startup
[39.692s] [ext: omni.isaac.core-1.24.3] startup
[39.937s] [ext: omni.isaac.synthetic_utils-0.3.5] startup
[39.942s] [ext: omni.kit.window.script_editor-1.6.2] startup
[39.949s] [ext: omni.kit.usd_undo-0.1.0] startup
[39.954s] [ext: semantics.schema.editor-0.2.3] startup
[39.963s] [ext: omni.physx.commands-1.4.15-5.1] startup
[39.974s] [ext: omni.kit.renderer.capture-0.0.0] startup
[39.982s] [ext: omni.physx.ui-1.4.15-5.1] startup
[40.078s] [ext: omni.warp-0.2.2] startup
Warp initialized:
   Version: 0.2.2
   CUDA device: NVIDIA RTX A6000
   Kernel cache: /root/.cache/warp/0.2.2
[40.508s] [ext: omni.kit.property.material-1.8.5] startup
[40.512s] [ext: omni.physx.demos-1.4.15-5.1] startup
[40.520s] [ext: omni.kit.window.toolbar-1.2.4] startup
[40.528s] [ext: omni.kit.property.physx-0.1.0] startup
2022-12-09 08:55:08 [40,674ms] [Warning] [omni.physx.plugin] Deprecated: getSimulationEventStream is deprecated, please use getSimulationEventStreamV2
[40.685s] [ext: omni.ui.scene-1.5.0] startup
[40.697s] [ext: omni.physx.tests-1.4.15-5.1] startup
[41.210s] [ext: omni.physx.vehicle-1.4.15-5.1] startup
[41.233s] [ext: omni.kit.manipulator.viewport-1.0.6] startup
[41.238s] [ext: omni.physx.camera-1.4.15-5.1] startup
[41.250s] [ext: omni.physx.cct-1.4.15-5.1] startup
[41.441s] [ext: omni.command.usd-1.0.1] startup
[41.450s] [ext: omni.kit.primitive.mesh-1.0.0] startup
[41.459s] [ext: omni.graph.visualization.nodes-1.1.1] startup
[41.465s] [ext: omni.physx.bundle-1.4.15-5.1] startup
[41.465s] [ext: omni.kit.widget.stage_icons-1.0.2] startup
[41.468s] [ext: omni.replicator.core-1.4.3] startup
[41.597s] [ext: omni.kit.widget.zoombar-1.0.3] startup
[41.599s] [ext: omni.rtx.window.settings-0.6.1] startup
[41.611s] [ext: omni.kit.window.stage-2.3.7] startup
[41.616s] [ext: omni.replicator.composer-1.1.3] startup
[41.625s] [ext: omni.kit.browser.core-2.0.12] startup
[41.635s] [ext: omni.rtx.settings.core-0.5.5] startup
[41.641s] [ext: omni.kit.window.extensions-1.1.0] startup
[41.650s] [ext: omni.kit.graph.usd.commands-1.1.0] startup
[41.656s] [ext: omni.kit.browser.folder.core-1.1.13] startup
[41.660s] [ext: omni.isaac.debug_draw-0.1.2] startup
[41.667s] [ext: omni.isaac.ui-0.2.1] startup
[41.672s] [ext: omni.kit.graph.delegate.default-1.0.15] startup
[41.673s] [ext: omni.isaac.motion_planning-0.2.0] startup
[41.681s] [ext: omni.isaac.lula-1.1.0] startup
[41.694s] [ext: omni.isaac.surface_gripper-0.1.2] startup
[41.701s] [ext: omni.kit.graph.editor.core-1.3.3] startup
[41.705s] [ext: omni.isaac.motion_generation-3.6.1] startup
[41.714s] [ext: omni.isaac.manipulators-1.0.1] startup
[41.717s] [ext: omni.isaac.core_nodes-0.13.0] startup
[41.734s] [ext: omni.kit.graph.widget.variables-2.0.2] startup
[41.737s] [ext: omni.kit.graph.delegate.modern-1.6.0] startup
[41.740s] [ext: omni.isaac.franka-0.3.0] startup
[41.741s] [ext: omni.isaac.wheeled_robots-0.5.8] startup
[41.766s] [ext: omni.graph.window.core-1.23.4] startup
[41.778s] [ext: omni.graph.instancing-1.1.4] startup
[41.784s] [ext: omni.graph.tutorials-1.1.2] startup
[41.808s] [ext: omni.graph.window.action-1.3.8] startup
[41.810s] [ext: omni.isaac.isaac_sensor-1.0.2] startup
2022-12-09 08:55:09 [41,811ms] [Warning] [omni.physx.plugin] Deprecated: getSimulationEventStream is deprecated, please use getSimulationEventStreamV2
[41.825s] [ext: omni.kit.selection-0.1.0] startup
[41.827s] [ext: omni.kit.widget.layers-1.5.17] startup
[41.850s] [ext: omni.graph.bundle.action-1.0.0] startup
[41.850s] [ext: omni.kit.profiler.window-1.4.4] startup
[41.863s] [ext: omni.kit.menu.edit-1.0.6] startup
[41.866s] [ext: omni.isaac.range_sensor-0.4.3] startup
[41.897s] [ext: omni.kit.property.layer-1.1.2] startup
[41.900s] [ext: omni.replicator.isaac-1.3.2] startup
[41.922s] [ext: omni.isaac.kit-0.2.1] startup
[41.924s] [ext: omni.graph.window.generic-1.3.8] startup
[41.926s] [ext: omni.isaac.utils-0.1.11] startup
[41.931s] [ext: omni.kit.property.audio-1.0.5] startup
[41.933s] [ext: omni.kit.property.skel-1.0.1] startup
[41.936s] [ext: omni.kit.property.render-1.1.0] startup
[41.937s] [ext: omni.kit.property.camera-1.0.3] startup
[41.939s] [ext: omni.kit.property.geometry-1.2.0] startup
[41.942s] [ext: omni.kit.property.light-1.0.5] startup
[41.944s] [ext: omni.kit.property.transform-1.0.2] startup
[41.948s] [ext: omni.isaac.universal_robots-0.3.0] startup
[41.950s] [ext: omni.isaac.occupancy_map-0.2.4] startup
[41.963s] [ext: omni.kit.window.console-0.2.0] startup
[41.969s] [ext: omni.kit.window.status_bar-0.1.1] startup
[41.974s] [ext: omni.kit.property.bundle-1.2.4] startup
[41.977s] [ext: omni.kit.menu.file-1.0.8] startup
[41.978s] [ext: omni.isaac.dofbot-0.3.0] startup
[41.980s] [ext: omni.kit.streamsdk.plugins-0.0.0] startup
[41.980s] [ext: omni.kit.livestream.core-0.0.0] startup
[41.985s] [ext: omni.kit.window.title-1.1.2] startup
2022-12-09 08:55:10 [41,979ms] [Error] [carb.windowing-glfw.plugin] GLFW initialization failed.
2022-12-09 08:55:10 [41,979ms] [Error] [carb] Failed to startup plugin carb.windowing-glfw.plugin (interfaces: [carb::windowing::IGLContext v1.0],[carb::windowing::IWindowing v1.3]) (impl: carb.windowing-glfw.plugin)
2022-12-09 08:55:10 [41,980ms] [Error] [carb.windowing-glfw.plugin] GLFW initialization failed.
2022-12-09 08:55:10 [41,980ms] [Error] [carb] Failed to startup plugin carb.windowing-glfw.plugin (interfaces: [carb::windowing::IGLContext v1.0],[carb::windowing::IWindowing v1.3]) (impl: carb.windowing-glfw.plugin)
[41.989s] [ext: omni.isaac.urdf-0.4.0] startup
[42.023s] [ext: omni.kit.widget.live-0.1.0] startup
2022-12-09 08:55:10 [42,018ms] [Warning] [omni.ext-live.plugin] Unable to detect Omniverse Cache Server. File /root/.nvidia-omniverse/config/Cache/Omniverse.toml is not found. Consider installing it (version >= 280) for better IO performance.
[42.027s] [ext: omni.kit.menu.common-1.0.0] startup
[42.030s] [ext: omni.kit.livestream.native-0.1.4] startup

Active user not found. Using default user [kiosk]Streaming server started.
[42.118s] [ext: omni.isaac.sim.python-2022.1.1] startup
[42.120s] Simulation App Starting
[42.935s] app ready
[46.282s] Simulation App Startup Complete
task_name: ${task.name}
experiment: 
num_envs: 
seed: 42
torch_deterministic: False
max_iterations: 
physics_engine: physx
pipeline: gpu
sim_device: gpu
device_id: 0
rl_device: cuda:0
num_threads: 4
solver_type: 1
test: False
checkpoint: 
headless: False
wandb_activate: False
wandb_group: 
wandb_name: test
wandb_entity: wazzup
wandb_project: isaactest
task: 
    name: FrankaCabinet
    physics_engine: ${..physics_engine}
    env: 
        numEnvs: 2
        envSpacing: 3.0
        episodeLength: 500
        enableDebugVis: False
        clipObservations: 5.0
        clipActions: 1.0
        controlFrequencyInv: 2
        startPositionNoise: 0.0
        startRotationNoise: 0.0
        numProps: 4
        aggregateMode: 3
        actionScale: 7.5
        dofVelocityScale: 0.1
        distRewardScale: 2.0
        rotRewardScale: 0.5
        aroundHandleRewardScale: 10.0
        openRewardScale: 7.5
        fingerDistRewardScale: 100.0
        actionPenaltyScale: 0.01
        fingerCloseRewardScale: 10.0
    sim: 
        dt: 0.0083
        use_gpu_pipeline: True
        gravity: [0.0, 0.0, -9.81]
        add_ground_plane: True
        use_flatcache: True
        enable_scene_query_support: False
        enable_cameras: False
        default_physics_material: 
            static_friction: 1.0
            dynamic_friction: 1.0
            restitution: 0.0
        physx: 
            worker_thread_count: 4
            solver_type: 1
            use_gpu: True
            solver_position_iteration_count: 12
            solver_velocity_iteration_count: 1
            contact_offset: 0.005
            rest_offset: 0.0
            bounce_threshold_velocity: 0.2
            friction_offset_threshold: 0.04
            friction_correlation_distance: 0.025
            enable_sleeping: True
            enable_stabilization: True
            max_depenetration_velocity: 1000.0
            gpu_max_rigid_contact_count: 524288
            gpu_max_rigid_patch_count: 33554432
            gpu_found_lost_pairs_capacity: 524288
            gpu_found_lost_aggregate_pairs_capacity: 262144
            gpu_total_aggregate_pairs_capacity: 1048576
            gpu_max_soft_body_contacts: 1048576
            gpu_max_particle_contacts: 1048576
            gpu_heap_capacity: 33554432
            gpu_temp_buffer_capacity: 16777216
            gpu_max_num_partitions: 8
        franka: 
            override_usd_defaults: False
            fixed_base: False
            enable_self_collisions: False
            enable_gyroscopic_forces: True
            solver_position_iteration_count: 12
            solver_velocity_iteration_count: 1
            sleep_threshold: 0.005
            stabilization_threshold: 0.001
            density: -1
            max_depenetration_velocity: 1000.0
            contact_offset: 0.005
            rest_offset: 0.0
        cabinet: 
            override_usd_defaults: False
            fixed_base: False
            enable_self_collisions: False
            enable_gyroscopic_forces: True
            solver_position_iteration_count: 12
            solver_velocity_iteration_count: 1
            sleep_threshold: 0.0
            stabilization_threshold: 0.001
            density: -1
            max_depenetration_velocity: 1000.0
            contact_offset: 0.005
            rest_offset: 0.0
        prop: 
            override_usd_defaults: False
            fixed_base: False
            enable_self_collisions: False
            enable_gyroscopic_forces: True
            solver_position_iteration_count: 12
            solver_velocity_iteration_count: 1
            sleep_threshold: 0.005
            stabilization_threshold: 0.001
            density: 100
            max_depenetration_velocity: 1000.0
            contact_offset: 0.005
            rest_offset: 0.0
train: 
    params: 
        seed: ${...seed}
        algo: 
            name: a2c_continuous
        model: 
            name: continuous_a2c_logstd
        network: 
            name: actor_critic
            separate: False
            space: 
                continuous: 
                    mu_activation: None
                    sigma_activation: None
                    mu_init: 
                        name: default
                    sigma_init: 
                        name: const_initializer
                        val: 0
                    fixed_sigma: True
            mlp: 
                units: [256, 128, 64]
                activation: elu
                d2rl: False
                initializer: 
                    name: default
                regularizer: 
                    name: None
        load_checkpoint: False
        load_path: None
        config: 
            name: FrankaCabinet
            full_experiment_name: test
            env_name: rlgpu
            device: cuda:0
            device_name: cuda:0
            ppo: True
            mixed_precision: False
            normalize_input: True
            normalize_value: True
            num_actors: 2
            reward_shaper: 
                scale_value: 0.01
            normalize_advantage: True
            gamma: 0.99
            tau: 0.95
            learning_rate: 5e-4
            lr_schedule: adaptive
            kl_threshold: 0.008
            score_to_win: 100000000
            max_epochs: 1500
            save_best_after: 200
            save_frequency: 100
            print_stats: True
            grad_norm: 1.0
            entropy_coef: 0.0
            truncate_grads: True
            e_clip: 0.2
            horizon_length: 16
            minibatch_size: 16
            mini_epochs: 8
            critic_coef: 4
            clip_value: True
            seq_len: 4
            bounds_loss_coef: 0.0001
name: FrankaCabinet
Sim params does not have attribute:  physx
Sim params does not have attribute:  franka
Sim params does not have attribute:  cabinet
Sim params does not have attribute:  prop
Pipeline:  GPU
Pipeline Device:  cuda:0
Sim Device:  GPU
Task Device: cuda:0
RL device:  cuda:0
/isaac-sim/exts/omni.isaac.ml_archive/pip_prebundle/gym/spaces/box.py:74: UserWarning: WARN: Box bound precision lowered by casting to float32
  "Box bound precision lowered by casting to {}".format(self.dtype)
[46.493s] [ext: omni.physx.flatcache-1.4.15-5.1] startup
2022-12-09 08:55:14 [46,522ms] [Warning] [carb] Acquiring non optional plugin interface which is not listed as dependency: [omni::kit::IStageUpdate v1.0] (plugin: (null)), by client: omni.physx.flatcache.plugin. Add it to CARB_PLUGIN_IMPL_DEPS() macro of a client.
2022-12-09 08:55:14 [46,571ms] [Warning] [omni.client.plugin]  Tick: authentication: Discovery(ws://localhost:3333): Error creating Api/Connection search: Not connected
2022-12-09 08:55:15 [47,748ms] [Warning] [omni.hydra.scene_delegate.plugin] Calling getBypassRenderSkelMeshProcessing for prim /World/envs/env_0/franka/panda_rightfinger/visuals.proto_visuals_id0 that has not been populated
2022-12-09 08:55:15 [47,798ms] [Warning] [omni.physx.plugin] ParseFEMClothMaterial: UsdPrim doesn't have a PhysxSchemaPhysxDeformableSurfaceMaterialAPI

gdb output:

Thread 1 "python" received signal SIGSEGV, Segmentation fault.
0x00007fa325614917 in ?? () from /isaac-sim/kit/extsPhysics/omni.physx-1.4.15-5.1/bin/libomni.physx.plugin.so
(gdb) bt
#0  0x00007fa325614917 in ?? () from /isaac-sim/kit/extsPhysics/omni.physx-1.4.15-5.1/bin/libomni.physx.plugin.so
#1  0x00007fa3256e899a in ?? () from /isaac-sim/kit/extsPhysics/omni.physx-1.4.15-5.1/bin/libomni.physx.plugin.so
#2  0x00007fa3256df6d8 in ?? () from /isaac-sim/kit/extsPhysics/omni.physx-1.4.15-5.1/bin/libomni.physx.plugin.so
#3  0x00007fa3257f85e5 in ?? () from /isaac-sim/kit/extsPhysics/omni.physx-1.4.15-5.1/bin/libomni.physx.plugin.so
#4  0x00007fa32572a9e2 in ?? () from /isaac-sim/kit/extsPhysics/omni.physx-1.4.15-5.1/bin/libomni.physx.plugin.so
#5  0x00007fa325136ed6 in ?? () from /isaac-sim/kit/extsPhysics/omni.physx-1.4.15-5.1/omni/physx/bindings/_physx.cpython-37m-x86_64-linux-gnu.so
#6  0x00007fa32513dde2 in ?? () from /isaac-sim/kit/extsPhysics/omni.physx-1.4.15-5.1/omni/physx/bindings/_physx.cpython-37m-x86_64-linux-gnu.so
#7  0x00000000004ba8f7 in _PyMethodDef_RawFastCallKeywords () at /usr/local/src/conda/python-3.7.15/Modules/opcode_targets.h:693
#8  0x00000000004ba6a6 in _PyCFunction_FastCallKeywords (func=func@entry=0x7fa3c4112fa0, args=args@entry=0x7f94c05a4a88, nargs=<optimized out>, kwnames=<optimized out>) at /usr/local/src/conda/python-3.7.15/Modules/opcode_targets.h:732
#9  0x00000000004c2c79 in _PyObject_FastCallKeywords () at /usr/local/src/conda/python-3.7.15/Modules/opcode_targets.h:156
#10 0x00000000004ba379 in call_function () at /usr/local/src/conda/python-3.7.15/Programs/gcmodule.c:4619
#11 0x00000000004b6cc2 in _PyEval_EvalFrameDefault () at /usr/local/src/conda/python-3.7.15/Programs/gcmodule.c:3093
#12 0x00000000004c4f56 in PyEval_EvalFrameEx (throwflag=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>, f=0x7f94c05a4910) at /usr/local/src/conda/python-3.7.15/Programs/gcmodule.c:544
#13 function_code_fastcall (globals=<optimized out>, nargs=1, nargs@entry=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>, args=<optimized out>, co=<optimized out>) at /usr/local/src/conda/python-3.7.15/Modules/opcode_targets.h:283
#14 _PyFunction_FastCallKeywords () at /usr/local/src/conda/python-3.7.15/Modules/opcode_targets.h:408
#15 0x00000000004ba23f in call_function () at /usr/local/src/conda/python-3.7.15/Programs/gcmodule.c:4616
#16 0x00000000004b310b in _PyEval_EvalFrameDefault () at /usr/local/src/conda/python-3.7.15/Programs/gcmodule.c:3110
#17 0x00000000004c4f56 in PyEval_EvalFrameEx (throwflag=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>, f=0x7f94c059d050) at /usr/local/src/conda/python-3.7.15/Programs/gcmodule.c:544
#18 function_code_fastcall (globals=<optimized out>, nargs=1, nargs@entry=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>, args=<optimized out>, co=<optimized out>) at /usr/local/src/conda/python-3.7.15/Modules/opcode_targets.h:283
#19 _PyFunction_FastCallKeywords () at /usr/local/src/conda/python-3.7.15/Modules/opcode_targets.h:408
#20 0x00000000004ba23f in call_function () at /usr/local/src/conda/python-3.7.15/Programs/gcmodule.c:4616
#21 0x00000000004b310b in _PyEval_EvalFrameDefault () at /usr/local/src/conda/python-3.7.15/Programs/gcmodule.c:3110
#22 0x00000000004c4f56 in PyEval_EvalFrameEx (throwflag=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>, f=0x7f94c0fbd810) at /usr/local/src/conda/python-3.7.15/Programs/gcmodule.c:544
#23 function_code_fastcall (globals=<optimized out>, nargs=1, nargs@entry=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>, args=<optimized out>, co=<optimized out>) at /usr/local/src/conda/python-3.7.15/Modules/opcode_targets.h:283
#24 _PyFunction_FastCallKeywords () at /usr/local/src/conda/python-3.7.15/Modules/opcode_targets.h:408
#25 0x00000000004ba23f in call_function () at /usr/local/src/conda/python-3.7.15/Programs/gcmodule.c:4616
#26 0x00000000004b6cc2 in _PyEval_EvalFrameDefault () at /usr/local/src/conda/python-3.7.15/Programs/gcmodule.c:3093
#27 0x00000000004b1411 in PyEval_EvalFrameEx (throwflag=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>, f=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>) at /usr/local/src/conda/python-3.7.15/Programs/gcmodule.c:547
#28 _PyEval_EvalCodeWithName () at /usr/local/src/conda/python-3.7.15/Programs/gcmodule.c:3930
#29 0x00000000004c50ec in _PyFunction_FastCallKeywords () at /usr/local/src/conda/python-3.7.15/Modules/opcode_targets.h:433
#30 0x00000000004ba23f in call_function () at /usr/local/src/conda/python-3.7.15/Programs/gcmodule.c:4616
#31 0x00000000004b3c41 in _PyEval_EvalFrameDefault () at /usr/local/src/conda/python-3.7.15/Programs/gcmodule.c:3139
#32 0x00000000004b1411 in PyEval_EvalFrameEx (throwflag=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>, f=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>) at /usr/local/src/conda/python-3.7.15/Programs/gcmodule.c:547
#33 _PyEval_EvalCodeWithName () at /usr/local/src/conda/python-3.7.15/Programs/gcmodule.c:3930
#34 0x00000000004c50ec in _PyFunction_FastCallKeywords () at /usr/local/src/conda/python-3.7.15/Modules/opcode_targets.h:433
#35 0x00000000004ba23f in call_function () at /usr/local/src/conda/python-3.7.15/Programs/gcmodule.c:4616
#36 0x00000000004b310b in _PyEval_EvalFrameDefault () at /usr/local/src/conda/python-3.7.15/Programs/gcmodule.c:3110
#37 0x00000000004b1411 in PyEval_EvalFrameEx (throwflag=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>, f=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>) at /usr/local/src/conda/python-3.7.15/Programs/gcmodule.c:547
#38 _PyEval_EvalCodeWithName () at /usr/local/src/conda/python-3.7.15/Programs/gcmodule.c:3930
#39 0x00000000004c50ec in _PyFunction_FastCallKeywords () at /usr/local/src/conda/python-3.7.15/Modules/opcode_targets.h:433
#40 0x00000000004ba23f in call_function () at /usr/local/src/conda/python-3.7.15/Programs/gcmodule.c:4616
#41 0x00000000004b6cc2 in _PyEval_EvalFrameDefault () at /usr/local/src/conda/python-3.7.15/Programs/gcmodule.c:3093
#42 0x00000000004b1411 in PyEval_EvalFrameEx (throwflag=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>, f=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>) at /usr/local/src/conda/python-3.7.15/Programs/gcmodule.c:547
#43 _PyEval_EvalCodeWithName () at /usr/local/src/conda/python-3.7.15/Programs/gcmodule.c:3930
#44 0x00000000004c50ec in _PyFunction_FastCallKeywords () at /usr/local/src/conda/python-3.7.15/Modules/opcode_targets.h:433
#45 0x00000000004ba23f in call_function () at /usr/local/src/conda/python-3.7.15/Programs/gcmodule.c:4616
#46 0x00000000004b3c41 in _PyEval_EvalFrameDefault () at /usr/local/src/conda/python-3.7.15/Programs/gcmodule.c:3139
#47 0x00000000004b1411 in PyEval_EvalFrameEx (throwflag=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>, f=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>) at /usr/local/src/conda/python-3.7.15/Programs/gcmodule.c:547
#48 _PyEval_EvalCodeWithName () at /usr/local/src/conda/python-3.7.15/Programs/gcmodule.c:3930
#49 0x00000000004c50ec in _PyFunction_FastCallKeywords () at /usr/local/src/conda/python-3.7.15/Modules/opcode_targets.h:433
#50 0x00000000004ba23f in call_function () at /usr/local/src/conda/python-3.7.15/Programs/gcmodule.c:4616
#51 0x00000000004b2fe1 in _PyEval_EvalFrameDefault () at /usr/local/src/conda/python-3.7.15/Programs/gcmodule.c:3124
#52 0x00000000004c4f56 in PyEval_EvalFrameEx (throwflag=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>, f=0x623c580) at /usr/local/src/conda/python-3.7.15/Programs/gcmodule.c:544
#53 function_code_fastcall (globals=<optimized out>, nargs=0, nargs@entry=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>, args=<optimized out>, co=<optimized out>) at /usr/local/src/conda/python-3.7.15/Modules/opcode_targets.h:283
#54 _PyFunction_FastCallKeywords () at /usr/local/src/conda/python-3.7.15/Modules/opcode_targets.h:408
---Type <return> to continue, or q <return> to quit---
#55 0x00000000004ba23f in call_function () at /usr/local/src/conda/python-3.7.15/Programs/gcmodule.c:4616
#56 0x00000000004b2fe1 in _PyEval_EvalFrameDefault () at /usr/local/src/conda/python-3.7.15/Programs/gcmodule.c:3124
#57 0x00000000004b1411 in PyEval_EvalFrameEx (throwflag=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>, f=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>) at /usr/local/src/conda/python-3.7.15/Programs/gcmodule.c:547
#58 _PyEval_EvalCodeWithName () at /usr/local/src/conda/python-3.7.15/Programs/gcmodule.c:3930
#59 0x00000000004b1209 in PyEval_EvalCodeEx () at /usr/local/src/conda/python-3.7.15/Programs/gcmodule.c:3959
#60 0x000000000054f39b in PyEval_EvalCode (co=co@entry=0x7fa8effb0ed0, globals=globals@entry=0x7fa8f0f7b370, locals=locals@entry=0x7fa8f0f7b370) at /usr/local/src/conda/python-3.7.15/Programs/gcmodule.c:524
#61 0x000000000056a263 in run_mod () at /usr/local/src/conda/python-3.7.15/Modules/Python-ast.c:1037
#62 0x0000000000570a17 in PyRun_FileExFlags () at /usr/local/src/conda/python-3.7.15/Modules/Python-ast.c:990
#63 0x000000000056ffd4 in PyRun_SimpleFileExFlags () at /usr/local/src/conda/python-3.7.15/Modules/Python-ast.c:429
#64 0x000000000056fb5b in PyRun_AnyFileExFlags () at /usr/local/src/conda/python-3.7.15/Modules/Python-ast.c:84
#65 0x0000000000544be3 in pymain_run_file (p_cf=0x7ffd63b5ade0, filename=<optimized out>, fp=0x259f7b0) at /tmp/build/80754af9/python_1669321496543/work/build-static/python.c:470
#66 pymain_run_filename (cf=0x7ffd63b5ade0, pymain=0x7ffd63b5aef0) at /tmp/build/80754af9/python_1669321496543/work/build-static/python.c:1660
#67 pymain_run_python (pymain=0x7ffd63b5aef0) at /tmp/build/80754af9/python_1669321496543/work/build-static/python.c:2970
#68 pymain_main () at /tmp/build/80754af9/python_1669321496543/work/build-static/python.c:3517
#69 0x000000000054487c in _Py_UnixMain (argc=<optimized out>, argv=<optimized out>) at /tmp/build/80754af9/python_1669321496543/work/build-static/python.c:3552
#70 0x00007fa8f0109c87 in __libc_start_main (main=0x544830 <main>, argc=5, argv=0x7ffd63b5b048, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7ffd63b5b038) at ../csu/libc-start.c:310
#71 0x000000000054472e in _start () at /usr/local/src/conda/python-3.7.15/Modules/pytime.c:2515

Any help would be much appreciated.

dHonerkamp commented 1 year ago

Apologies, due to a bug I was running our custom robot, not the example one