CUDA error: device-side assert triggered

[ChengEeee]

" File "train.py", line 66, in train ppo_runner.learn(num_learning_iterations=train_cfg.runner.max_iterations, init_at_random_ep_len=True) File "/home/hello/Aaron_Project/extreme-parkour/rsl_rl/rsl_rl/runners/on_policy_runner.py", line 165, in learn_RL obs, privileged_obs, rewards, dones, infos = self.env.step(actions) # obs has changed to next_obs !! if done obs has been reset File "/home/hello/Aaron_Project/extreme-parkour/legged_gym/legged_gym/envs/base/legged_robot.py", line 145, in step self.post_physics_step() File "/home/hello/Aaron_Project/extreme-parkour/legged_gym/legged_gym/envs/base/legged_robot.py", line 259, in post_physics_step self.reset_idx(env_ids) File "/home/hello/Aaron_Project/extreme-parkour/legged_gym/legged_gym/envs/base/legged_robot.py", line 327, in reset_idx self._reset_dofs(env_ids) File "/home/hello/Aaron_Project/extreme-parkour/legged_gym/legged_gym/envs/base/legged_robot.py", line 636, in _reset_dofs self.dof_vel[env_ids] = 0. RuntimeError: CUDA error: device-side assert triggered CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect. For debugging consider passing CUDA_LAUNCH_BLOCKING=1." i try to train base_policy,training is normal for the first few thousand rounds,but the error will occur every time i train for less than 10,000 rounds. Looking forward to your reply，thanks

[geyang]

@ChengEeee how did you fix this?