search

isaac-sim / IsaacLab

Unified framework for robot learning built on NVIDIA Isaac Sim
https://isaac-sim.github.io/IsaacLab
Other
2.09k stars 844 forks source link

[Bug Report] Error using MultiAssetSpawnerCfg #1258

Open chandramohanjagadeesan opened 1 week ago

chandramohanjagadeesan commented 1 week ago

Describe the bug

I used MultiAssetSpawnerCfg to import objects from usd file. When the environment gets reset in super()._reset_idx(env_ids). there is a following error. This error occurs in reset in directRL environment

Steps to reproduce

Cfg in the interactive sceneCfg

object: RigidObjectCfg = RigidObjectCfg( primpath="/World/envs/env.*/Object", spawn=sim_utils.MultiAssetSpawnerCfg( assets_cfg=[ sim_utils.UsdFileCfg( usd_path=f"{dir_path}/Assets/Objects/USD/object_0.usd", scale = [1.0, 1.0, 1.0], ), sim_utils.UsdFileCfg( usd_path=f"{dir_path}/Assets/Objects/USD/object_1.usd", scale = [1.0, 1.0, 1.0], ), sim_utils.UsdFileCfg( usd_path=f"{dir_path}/Assets/Objects/USD/object_2.usd", scale = [1.0, 1.0, 1.0], ), sim_utils.UsdFileCfg( usd_path=f"{dir_path}/Assets/Objects/USD/object_3.usd", scale = [1.0, 1.0, 1.0], ) ], random_choice=True, rigid_props=sim_utils.RigidBodyPropertiesCfg( solver_position_iteration_count=4, solver_velocity_iteration_count=0 ), mass_props=sim_utils.MassPropertiesCfg(mass=1.0), collision_props=sim_utils.CollisionPropertiesCfg(), ), init_state=RigidObjectCfg.InitialStateCfg(pos=(0.5, 0.0, 0.0), rot=(0.0, -0.0, 0.0, -0.0)), )

Error code

Exception has occurred: RuntimeError CUDA error: device-side assert triggered CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect. For debugging consider passing CUDA_LAUNCH_BLOCKING=1 Compile with TORCH_USE_CUDA_DSA to enable device-side assertions. File "/media/team_omd/DATA/omniverse/pkg/IsaacLab/IsaacLab/source/extensions/omni.isaac.lab/omni/isaac/lab/assets/rigid_object/rigid_object.py", line 104, in reset self._external_torque_b[env_ids] = 0.0 File "/media/team_omd/DATA/omniverse/pkg/IsaacLab/IsaacLab/source/extensions/omni.isaac.lab/omni/isaac/lab/scene/interactive_scene.py", line 351, in reset rigid_object.reset(env_ids) File "/media/team_omd/DATA/omniverse/pkg/IsaacLab/IsaacLab/source/extensions/omni.isaac.lab/omni/isaac/lab/envs/direct_rl_env.py", line 537, in _reset_idx self.scene.reset(env_ids) File "/home/team_omd/Projects/chandra_ma/continual_nbv/NBVPlanning/Environment/NBVEnv.py", line 430, in _reset_idx super()._reset_idx(env_ids) File "/media/team_omd/DATA/omniverse/pkg/IsaacLab/IsaacLab/source/extensions/omni.isaac.lab/omni/isaac/lab/envs/direct_rl_env.py", line 271, in reset self._reset_idx(indices) File "/media/team_omd/DATA/omniverse/pkg/IsaacLab/IsaacLab/source/extensions/omni.isaac.lab_tasks/omni/isaac/lab_tasks/utils/wrappers/sb3.py", line 207, in reset obsdict, = self.env.reset() File "/home/team_omd/Projects/chandra_ma/continual_nbv/NBVPlanning/test.py", line 106, in main obs = env.reset() File "/home/team_omd/Projects/chandra_ma/continual_nbv/NBVPlanning/test.py", line 138, in main() RuntimeError: CUDA error: device-side assert triggered CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect. For debugging consider passing CUDA_LAUNCH_BLOCKING=1 Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.

System Info

Describe the characteristic of your environment:

Additional context

Checklist

Acceptance Criteria

Add the criteria for which this task is considered done. If not known at issue creation time, you can add this once the issue is assigned.

TuluhanAkbulut commented 1 week ago

I am getting the same error with the ManagerbasedRL environments too. Here, if I spawn using the following path usd_path=[

# self.asset_config_path + "Blocks/DexCube/dex_cube_instanceable.usd",
# self.asset_config_path + "Blocks/blue_block.usd",
# self.asset_config_path + "Blocks/green_block.usd",
# self.asset_config_path + "Blocks/red_block.usd",
# self.asset_config_path + "Blocks/yellow_block.usd",
# self.asset_config_path + "Mugs/SM_Mug_A2.usd",
# self.asset_config_path + "Mugs/SM_Mug_B1.usd",

],

If usd path consists of red, blue, green, and yellow blocks, it works fine. If dex_cube is alone in the list, it works fine too. However, if it consists dex_cube with other blocks it throws the CUDA error. For SM_Mugs it is saying that AttributeError: 'RigidObject' object has no attribute '_external_force_b' which may be unrelated.

chandramohanjagadeesan commented 1 week ago

Hi @TuluhanAkbulut

The issue is in reset method in RigidObject.py file.

It is caused by how self._external_force_b is indexed with env_ids to set value 0.0. I found a work around to make it work. You can change the lines

reset external wrench

    self._external_force_b[env_ids] = 0.0
    self._external_torque_b[env_ids] = 0.0

to

reset external wrench

    self._external_force_b[0,0,env_ids] = 0.0
    self._external_torque_b[0,0,env_ids] = 0.0

This makes it work but I have to test if this change has any implications on other functions.

Error occurs because env_ids is of shape tensor([3]) and self._external_force_b[env_ids] is of shape tensor[1,1,3]. Hope this helps till the actual solution comes up.

For note : @Mayankm96. I am tagging you because I found that you developed this feature from the past issues and I would like to know if this solution would work. Thanks in advance for your inputs.

TuluhanAkbulut commented 5 days ago

Thanks for the reply, it did not work for me. It actually made it worse by throwing the same error even for only blue_block in the list, which was not happening before. Here, the related code and error message:

I am trying this method with Omniverse-nucleus objects at the moment:

self.asset_config_path = http://omniverse-content-production.s3-us-west-2.amazonaws.com/Assets/Isaac/4.2/Isaac/Props/

self.scene.object2 = RigidObjectCfg(
    prim_path="{ENV_REGEX_NS}/Object2",
    spawn=sim_utils.MultiAssetSpawnerCfg(
    assets_cfg=[
        # sim_utils.UsdFileCfg(
        # usd_path=self.asset_config_path + "Blocks/DexCube/dex_cube_instanceable.usd",
        # scale = [0.8, 0.8, 0.8],
        # ),
        sim_utils.UsdFileCfg(
        usd_path=self.asset_config_path + "Blocks/blue_block.usd",
        scale = [1.0, 1.0, 1.0],
        ),
        # sim_utils.UsdFileCfg(
        # usd_path=self.asset_config_path + "Blocks/red_block.usd",
        # scale = [0.7, 0.7, 0.7],
        # ),
    ],
    random_choice=False,
    rigid_props=sim_utils.RigidBodyPropertiesCfg(
        solver_position_iteration_count=4, solver_velocity_iteration_count=0
    ),
    mass_props=sim_utils.MassPropertiesCfg(mass=1.0),
    collision_props=sim_utils.CollisionPropertiesCfg(),
),
init_state=RigidObjectCfg.InitialStateCfg(pos=[0.8, 0.2, 0.1], rot=[1, 0, 0, 0]),
)

Error message:

../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [3,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed. ../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [4,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed. ../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [5,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed. ../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [6,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed. ../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [7,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed. ../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [8,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed. ../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [9,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed. ../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [10,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed. ../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [11,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed. ../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [12,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed. ../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [13,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed. ../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [14,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed. ../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [15,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed. ../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [16,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed. ../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [17,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed. ../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [18,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed. ../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [19,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed. ../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [20,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed. ../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [21,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed. ../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [22,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed. ../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [23,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed. ../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [24,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed. ../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [25,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed. ../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [26,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed. ../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [27,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed. ../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [28,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed. ../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [29,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed. ../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [30,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed. ../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [31,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed. ../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [32,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed. ../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [33,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed. ../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [34,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed. ../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [35,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed. ../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [36,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed. ../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [37,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed. ../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [38,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed. ../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [39,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed. ../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [40,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed. ../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [41,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed. ../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [42,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed. ../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [43,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed. ../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [44,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed. ../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [45,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed. ../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [46,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed. ../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [47,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed. ../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [48,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed. ../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [49,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed. ../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [50,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed. ../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [51,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed. ../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [52,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed. ../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [53,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed. ../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [54,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed. ../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [55,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed. ../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [56,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed. ../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [57,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed. ../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [58,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed. ../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [59,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed. ../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [60,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed. ../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [61,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed. ../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [62,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed. ../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [63,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed. ../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [64,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed. ../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [65,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed. ../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [66,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed. ../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [67,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed. ../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [68,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed. ../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [69,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed. ../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [70,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed. ../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [71,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed. ../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [72,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed. ../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [73,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed. ../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [74,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed. ../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [75,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed. ../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [76,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed. ../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [77,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed. ../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [78,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed. ../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [79,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed. ../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [80,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed. ../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [81,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed. ../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [82,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed. ../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [83,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed. ../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [84,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed. ../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [85,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed. ../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [86,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed. ../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [87,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed. ../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [88,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed. ../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [89,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed. ../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [90,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed. ../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [91,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed. ../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [92,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed. ../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [93,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed. ../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [94,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed. ../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [95,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed. Traceback (most recent call last): File "/home/tuluhan/Desktop/Concept/isaac_ws/simple.py", line 102, in main() File "/home/tuluhan/Desktop/Concept/isaac_ws/simple.py", line 86, in main env.reset() File "/home/tuluhan/miniconda3/envs/isaac_env/lib/python3.10/site-packages/gymnasium/wrappers/order_enforcing.py", line 61, in reset return self.env.reset(**kwargs) File "/home/tuluhan/Desktop/IsaacLab/source/extensions/omni.isaac.lab/omni/isaac/lab/envs/manager_based_env.py", line 252, in reset self._reset_idx(indices) File "/home/tuluhan/Desktop/IsaacLab/source/extensions/omni.isaac.lab/omni/isaac/lab/envs/manager_based_rl_env.py", line 325, in _reset_idx self.scene.reset(env_ids) File "/home/tuluhan/Desktop/IsaacLab/source/extensions/omni.isaac.lab/omni/isaac/lab/scene/interactive_scene.py", line 351, in reset rigid_object.reset(env_ids) File "/home/tuluhan/Desktop/IsaacLab/source/extensions/omni.isaac.lab/omni/isaac/lab/assets/rigid_object/rigid_object.py", line 103, in reset self._external_force_b[0, env_ids] = 0.0 RuntimeError: CUDA error: device-side assert triggered Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.

2024-10-21 01:51:00 [6,277ms] [Error] [omni.physx.fabric.plugin] CUDA error: device-side assert triggered: ../../../extensions/runtime/source/omni.physx.fabric/plugins/DirectGpuHelper.cpp: 328 2024-10-21 01:51:00 [6,277ms] [Error] [omni.physx.fabric.plugin] CUDA error: device-side assert triggered: ../../../extensions/runtime/source/omni.physx.fabric/plugins/DirectGpuHelper.cpp: 331 2024-10-21 01:51:00 [6,277ms] [Error] [omni.physx.fabric.plugin] CUDA error: device-side assert triggered: ../../../extensions/runtime/source/omni.physx.fabric/plugins/DirectGpuHelper.cpp: 334 2024-10-21 01:51:00 [6,277ms] [Error] [omni.physx.fabric.plugin] CUDA error: device-side assert triggered: ../../../extensions/runtime/source/omni.physx.fabric/plugins/DirectGpuHelper.cpp: 337 2024-10-21 01:51:00 [6,277ms] [Error] [omni.physx.fabric.plugin] CUDA error: device-side assert triggered: ../../../extensions/runtime/source/omni.physx.fabric/plugins/DirectGpuHelper.cpp: 340 2024-10-21 01:51:00 [6,278ms] [Warning] [omni.physx.plugin] PhysX warning: /builds/omniverse/physics/physx/source/gpucommon/src/PxgCudaMemoryAllocator.cpp, FILE /builds/omniverse/physics/physx/source/gpucommon/src/PxgCudaMemoryAllocator.cpp, LINE 167 2024-10-21 01:51:00 [6,279ms] [Warning] [omni.physx.plugin] PhysX warning: /builds/omniverse/physics/physx/source/gpucommon/src/PxgCudaMemoryAllocator.cpp, FILE /builds/omniverse/physics/physx/source/gpucommon/src/PxgCudaMemoryAllocator.cpp, LINE 167 2024-10-21 01:51:00 [6,279ms] [Warning] [omni.physx.plugin] PhysX warning: /builds/omniverse/physics/physx/source/gpucommon/src/PxgCudaMemoryAllocator.cpp, FILE /builds/omniverse/physics/physx/source/gpucommon/src/PxgCudaMemoryAllocator.cpp, LINE 167 2024-10-21 01:51:00 [6,279ms] [Warning] [omni.physx.plugin] PhysX warning: /builds/omniverse/physics/physx/source/gpucommon/src/PxgCudaMemoryAllocator.cpp, FILE /builds/omniverse/physics/physx/source/gpucommon/src/PxgCudaMemoryAllocator.cpp, LINE 167 2024-10-21 01:51:00 [6,279ms] [Warning] [omni.physx.plugin] PhysX warning: /builds/omniverse/physics/physx/source/gpucommon/src/PxgCudaMemoryAllocator.cpp, FILE /builds/omniverse/physics/physx/source/gpucommon/src/PxgCudaMemoryAllocator.cpp, LINE 167 2024-10-21 01:51:00 [6,279ms] [Warning] [omni.physx.plugin] PhysX warning: /builds/omniverse/physics/physx/source/gpucommon/src/PxgCudaMemoryAllocator.cpp, FILE /builds/omniverse/physics/physx/source/gpucommon/src/PxgCudaMemoryAllocator.cpp, LINE 167 2024-10-21 01:51:00 [6,281ms] [Error] [omni.physx.plugin] PhysX error: Failed to unload CUDA module data, returned 710., FILE /builds/omniverse/physics/physx/source/cudamanager/src/CudaContextManager.cpp, LINE 817 2024-10-21 01:51:00 [6,281ms] [Error] [omni.physx.plugin] PhysX error: Failed to unload CUDA module data, returned 710., FILE /builds/omniverse/physics/physx/source/cudamanager/src/CudaContextManager.cpp, LINE 817 2024-10-21 01:51:00 [6,281ms] [Error] [omni.physx.plugin] PhysX error: Failed to unload CUDA module data, returned 710., FILE /builds/omniverse/physics/physx/source/cudamanager/src/CudaContextManager.cpp, LINE 817 2024-10-21 01:51:00 [6,281ms] [Error] [omni.physx.plugin] PhysX error: Failed to unload CUDA module data, returned 710., FILE /builds/omniverse/physics/physx/source/cudamanager/src/CudaContextManager.cpp, LINE 817 2024-10-21 01:51:00 [6,281ms] [Error] [omni.physx.plugin] PhysX error: Failed to unload CUDA module data, returned 710., FILE /builds/omniverse/physics/physx/source/cudamanager/src/CudaContextManager.cpp, LINE 817 2024-10-21 01:51:00 [6,281ms] [Error] [omni.physx.plugin] PhysX error: Failed to unload CUDA module data, returned 710., FILE /builds/omniverse/physics/physx/source/cudamanager/src/CudaContextManager.cpp, LINE 817 2024-10-21 01:51:00 [6,281ms] [Error] [omni.physx.plugin] PhysX error: Failed to unload CUDA module data, returned 710., FILE /builds/omniverse/physics/physx/source/cudamanager/src/CudaContextManager.cpp, LINE 817 2024-10-21 01:51:00 [6,282ms] [Error] [omni.physx.plugin] PhysX error: Failed to unload CUDA module data, returned 710., FILE /builds/omniverse/physics/physx/source/cudamanager/src/CudaContextManager.cpp, LINE 817 2024-10-21 01:51:00 [6,282ms] [Error] [omni.physx.plugin] PhysX error: Failed to unload CUDA module data, returned 710., FILE /builds/omniverse/physics/physx/source/cudamanager/src/CudaContextManager.cpp, LINE 817 2024-10-21 01:51:00 [6,282ms] [Error] [omni.physx.plugin] PhysX error: Failed to unload CUDA module data, returned 710., FILE /builds/omniverse/physics/physx/source/cudamanager/src/CudaContextManager.cpp, LINE 817 2024-10-21 01:51:00 [6,282ms] [Error] [omni.physx.plugin] PhysX error: Failed to unload CUDA module data, returned 710., FILE /builds/omniverse/physics/physx/source/cudamanager/src/CudaContextManager.cpp, LINE 817 2024-10-21 01:51:00 [6,282ms] [Error] [omni.physx.plugin] PhysX error: Failed to unload CUDA module data, returned 710., FILE /builds/omniverse/physics/physx/source/cudamanager/src/CudaContextManager.cpp, LINE 817 2024-10-21 01:51:00 [6,282ms] [Error] [omni.physx.plugin] PhysX error: Failed to unload CUDA module data, returned 710., FILE /builds/omniverse/physics/physx/source/cudamanager/src/CudaContextManager.cpp, LINE 817 2024-10-21 01:51:00 [6,282ms] [Error] [omni.physx.plugin] PhysX error: Failed to unload CUDA module data, returned 710., FILE /builds/omniverse/physics/physx/source/cudamanager/src/CudaContextManager.cpp, LINE 817 2024-10-21 01:51:00 [6,282ms] [Error] [omni.physx.plugin] PhysX error: Failed to unload CUDA module data, returned 710., FILE /builds/omniverse/physics/physx/source/cudamanager/src/CudaContextManager.cpp, LINE 817 2024-10-21 01:51:00 [6,282ms] [Error] [omni.physx.plugin] PhysX error: Failed to unload CUDA module data, returned 710., FILE /builds/omniverse/physics/physx/source/cudamanager/src/CudaContextManager.cpp, LINE 817 2024-10-21 01:51:00 [6,283ms] [Error] [omni.physx.plugin] PhysX error: Failed to unload CUDA module data, returned 710., FILE /builds/omniverse/physics/physx/source/cudamanager/src/CudaContextManager.cpp, LINE 817 2024-10-21 01:51:00 [6,283ms] [Error] [omni.physx.plugin] PhysX error: Failed to unload CUDA module data, returned 710., FILE /builds/omniverse/physics/physx/source/cudamanager/src/CudaContextManager.cpp, LINE 817 2024-10-21 01:51:00 [6,283ms] [Error] [omni.physx.plugin] PhysX error: Failed to unload CUDA module data, returned 710., FILE /builds/omniverse/physics/physx/source/cudamanager/src/CudaContextManager.cpp, LINE 817 2024-10-21 01:51:00 [6,283ms] [Error] [omni.physx.plugin] PhysX error: Failed to unload CUDA module data, returned 710., FILE /builds/omniverse/physics/physx/source/cudamanager/src/CudaContextManager.cpp, LINE 817 2024-10-21 01:51:00 [6,283ms] [Error] [omni.physx.plugin] PhysX error: Failed to unload CUDA module data, returned 710., FILE /builds/omniverse/physics/physx/source/cudamanager/src/CudaContextManager.cpp, LINE 817 2024-10-21 01:51:00 [6,283ms] [Error] [omni.physx.plugin] PhysX error: Failed to unload CUDA module data, returned 710., FILE /builds/omniverse/physics/physx/source/cudamanager/src/CudaContextManager.cpp, LINE 817 2024-10-21 01:51:00 [6,284ms] [Error] [omni.physx.plugin] PhysX error: Failed to unload CUDA module data, returned 710., FILE /builds/omniverse/physics/physx/source/cudamanager/src/CudaContextManager.cpp, LINE 817 2024-10-21 01:51:00 [6,284ms] [Error] [omni.physx.plugin] PhysX error: Failed to unload CUDA module data, returned 710., FILE /builds/omniverse/physics/physx/source/cudamanager/src/CudaContextManager.cpp, LINE 817 2024-10-21 01:51:00 [6,284ms] [Error] [omni.physx.plugin] PhysX error: Failed to unload CUDA module data, returned 710., FILE /builds/omniverse/physics/physx/source/cudamanager/src/CudaContextManager.cpp, LINE 817 2024-10-21 01:51:00 [6,284ms] [Error] [omni.physx.plugin] PhysX error: Failed to unload CUDA module data, returned 710., FILE /builds/omniverse/physics/physx/source/cudamanager/src/CudaContextManager.cpp, LINE 817 2024-10-21 01:51:00 [6,284ms] [Error] [omni.physx.plugin] PhysX error: Failed to unload CUDA module data, returned 710., FILE /builds/omniverse/physics/physx/source/cudamanager/src/CudaContextManager.cpp, LINE 817 2024-10-21 01:51:00 [6,284ms] [Error] [omni.physx.plugin] PhysX error: Failed to unload CUDA module data, returned 710., FILE /builds/omniverse/physics/physx/source/cudamanager/src/CudaContextManager.cpp, LINE 817 2024-10-21 01:51:00 [6,284ms] [Error] [omni.physx.plugin] PhysX error: Failed to unload CUDA module data, returned 710., FILE /builds/omniverse/physics/physx/source/cudamanager/src/CudaContextManager.cpp, LINE 817 2024-10-21 01:51:00 [6,284ms] [Error] [omni.physx.plugin] PhysX error: Failed to unload CUDA module data, returned 710., FILE /builds/omniverse/physics/physx/source/cudamanager/src/CudaContextManager.cpp, LINE 817 2024-10-21 01:51:00 [6,284ms] [Error] [omni.physx.plugin] PhysX error: Failed to unload CUDA module data, returned 710., FILE /builds/omniverse/physics/physx/source/cudamanager/src/CudaContextManager.cpp, LINE 817 2024-10-21 01:51:00 [6,285ms] [Error] [omni.physx.plugin] PhysX error: Failed to unload CUDA module data, returned 710., FILE /builds/omniverse/physics/physx/source/cudamanager/src/CudaContextManager.cpp, LINE 817 2024-10-21 01:51:00 [6,285ms] [Error] [omni.physx.plugin] PhysX error: Failed to unload CUDA module data, returned 710., FILE /builds/omniverse/physics/physx/source/cudamanager/src/CudaContextManager.cpp, LINE 817 2024-10-21 01:51:00 [6,285ms] [Error] [omni.physx.plugin] PhysX error: Failed to unload CUDA module data, returned 710., FILE /builds/omniverse/physics/physx/source/cudamanager/src/CudaContextManager.cpp, LINE 817 2024-10-21 01:51:00 [6,285ms] [Error] [omni.physx.plugin] PhysX error: Failed to unload CUDA module data, returned 710., FILE /builds/omniverse/physics/physx/source/cudamanager/src/CudaContextManager.cpp, LINE 817 2024-10-21 01:51:00 [6,285ms] [Error] [omni.physx.plugin] PhysX error: Failed to unload CUDA module data, returned 710., FILE /builds/omniverse/physics/physx/source/cudamanager/src/CudaContextManager.cpp, LINE 817 2024-10-21 01:51:00 [6,285ms] [Error] [omni.physx.plugin] PhysX error: Failed to unload CUDA module data, returned 710., FILE /builds/omniverse/physics/physx/source/cudamanager/src/CudaContextManager.cpp, LINE 817 2024-10-21 01:51:00 [6,285ms] [Error] [omni.physx.plugin] PhysX error: Failed to unload CUDA module data, returned 710., FILE /builds/omniverse/physics/physx/source/cudamanager/src/CudaContextManager.cpp, LINE 817 2024-10-21 01:51:00 [6,285ms] [Error] [omni.physx.plugin] PhysX error: Failed to unload CUDA module data, returned 710., FILE /builds/omniverse/physics/physx/source/cudamanager/src/CudaContextManager.cpp, LINE 817 2024-10-21 01:51:00 [6,285ms] [Error] [omni.physx.plugin] PhysX error: Failed to unload CUDA module data, returned 710., FILE /builds/omniverse/physics/physx/source/cudamanager/src/CudaContextManager.cpp, LINE 817 2024-10-21 01:51:00 [6,286ms] [Error] [omni.physx.plugin] PhysX error: Failed to unload CUDA module data, returned 710., FILE /builds/omniverse/physics/physx/source/cudamanager/src/CudaContextManager.cpp, LINE 817 2024-10-21 01:51:00 [6,286ms] [Error] [omni.physx.plugin] PhysX error: Failed to unload CUDA module data, returned 710., FILE /builds/omniverse/physics/physx/source/cudamanager/src/CudaContextManager.cpp, LINE 817 2024-10-21 01:51:00 [6,286ms] [Error] [omni.physx.plugin] PhysX error: Failed to unload CUDA module data, returned 710., FILE /builds/omniverse/physics/physx/source/cudamanager/src/CudaContextManager.cpp, LINE 817 2024-10-21 01:51:00 [6,286ms] [Error] [omni.physx.plugin] PhysX error: Failed to unload CUDA module data, returned 710., FILE /builds/omniverse/physics/physx/source/cudamanager/src/CudaContextManager.cpp, LINE 817 2024-10-21 01:51:00 [6,286ms] [Error] [omni.physx.plugin] PhysX error: Failed to unload CUDA module data, returned 710., FILE /builds/omniverse/physics/physx/source/cudamanager/src/CudaContextManager.cpp, LINE 817 2024-10-21 01:51:00 [6,286ms] [Error] [omni.physx.plugin] PhysX error: Failed to unload CUDA module data, returned 710., FILE /builds/omniverse/physics/physx/source/cudamanager/src/CudaContextManager.cpp, LINE 817 2024-10-21 01:51:00 [6,286ms] [Error] [omni.physx.plugin] PhysX error: Failed to unload CUDA module data, returned 710., FILE /builds/omniverse/physics/physx/source/cudamanager/src/CudaContextManager.cpp, LINE 817 2024-10-21 01:51:00 [6,286ms] [Error] [omni.physx.plugin] PhysX error: Failed to unload CUDA module data, returned 710., FILE /builds/omniverse/physics/physx/source/cudamanager/src/CudaContextManager.cpp, LINE 817 2024-10-21 01:51:00 [6,287ms] [Error] [omni.physx.plugin] PhysX error: Failed to unload CUDA module data, returned 710., FILE /builds/omniverse/physics/physx/source/cudamanager/src/CudaContextManager.cpp, LINE 817 2024-10-21 01:51:00 [6,287ms] [Error] [omni.physx.plugin] PhysX error: Failed to unload CUDA module data, returned 710., FILE /builds/omniverse/physics/physx/source/cudamanager/src/CudaContextManager.cpp, LINE 817 2024-10-21 01:51:00 [6,287ms] [Error] [omni.physx.plugin] PhysX error: Failed to unload CUDA module data, returned 710., FILE /builds/omniverse/physics/physx/source/cudamanager/src/CudaContextManager.cpp, LINE 817 2024-10-21 01:51:00 [6,287ms] [Error] [omni.physx.plugin] PhysX error: Failed to unload CUDA module data, returned 710., FILE /builds/omniverse/physics/physx/source/cudamanager/src/CudaContextManager.cpp, LINE 817 2024-10-21 01:51:00 [6,287ms] [Error] [omni.physx.plugin] PhysX error: Failed to unload CUDA module data, returned 710., FILE /builds/omniverse/physics/physx/source/cudamanager/src/CudaContextManager.cpp, LINE 817 2024-10-21 01:51:00 [6,287ms] [Error] [omni.physx.plugin] PhysX error: Failed to unload CUDA module data, returned 710., FILE /builds/omniverse/physics/physx/source/cudamanager/src/CudaContextManager.cpp, LINE 817 2024-10-21 01:51:00 [6,287ms] [Error] [omni.physx.plugin] PhysX error: Failed to unload CUDA module data, returned 710., FILE /builds/omniverse/physics/physx/source/cudamanager/src/CudaContextManager.cpp, LINE 817 2024-10-21 01:51:00 [6,288ms] [Error] [omni.physx.plugin] PhysX error: Failed to unload CUDA module data, returned 710., FILE /builds/omniverse/physics/physx/source/cudamanager/src/CudaContextManager.cpp, LINE 817 2024-10-21 01:51:00 [6,288ms] [Error] [omni.physx.plugin] PhysX error: Failed to unload CUDA module data, returned 710., FILE /builds/omniverse/physics/physx/source/cudamanager/src/CudaContextManager.cpp, LINE 817 2024-10-21 01:51:00 [6,288ms] [Error] [omni.physx.plugin] PhysX error: Failed to unload CUDA module data, returned 710., FILE /builds/omniverse/physics/physx/source/cudamanager/src/CudaContextManager.cpp, LINE 817 2024-10-21 01:51:00 [6,288ms] [Error] [omni.physx.plugin] PhysX error: Failed to unload CUDA module data, returned 710., FILE /builds/omniverse/physics/physx/source/cudamanager/src/CudaContextManager.cpp, LINE 817 2024-10-21 01:51:00 [6,288ms] [Error] [omni.physx.plugin] PhysX error: Failed to unload CUDA module data, returned 710., FILE /builds/omniverse/physics/physx/source/cudamanager/src/CudaContextManager.cpp, LINE 817 2024-10-21 01:51:00 [6,289ms] [Error] [omni.physx.plugin] PhysX error: Failed to unload CUDA module data, returned 710., FILE /builds/omniverse/physics/physx/source/cudamanager/src/CudaContextManager.cpp, LINE 817 2024-10-21 01:51:00 [6,289ms] [Error] [omni.physx.plugin] PhysX error: Failed to unload CUDA module data, returned 710., FILE /builds/omniverse/physics/physx/source/cudamanager/src/CudaContextManager.cpp, LINE 817