RuntimeError: Failed to find forward kernel

[curieuxjy]

1. cuRobo installation mode (choose from [python, isaac sim, docker python, docker isaac sim]): isaac sim
2. python version: use omni_python in isaac-sim 2023.1.1
3. Isaac Sim version (if using): 2023.1.1

---

Issue Details

I tried to run some examples/isaac_sim codes.

When running multi_arm_reacher.py, I got this error:

RuntimeError: Failed to find forward kernel 'forward_l2_loop12' from module 'curobo.rollout.cost.dist_cost' for device 'cuda:0'

Is there any solution for this?

this is a whole error message:

Traceback (most recent call last):
  File "/home/avery/Documents/curobo/examples/isaac_sim/multi_arm_reacher.py", line 326, in <module>
    main()
  File "/home/avery/Documents/curobo/examples/isaac_sim/multi_arm_reacher.py", line 123, in main
    motion_gen.warmup(enable_graph=True, warmup_js_trajopt=False)
  File "/home/avery/Documents/curobo/src/curobo/wrap/reacher/motion_gen.py", line 1887, in warmup
    self.plan_single(
  File "/home/avery/Documents/curobo/src/curobo/wrap/reacher/motion_gen.py", line 1528, in plan_single
    result = self._plan_attempts(
  File "/home/avery/Documents/curobo/src/curobo/wrap/reacher/motion_gen.py", line 2963, in _plan_attempts
    result = self._plan_from_solve_state(
  File "/home/avery/Documents/curobo/src/curobo/wrap/reacher/motion_gen.py", line 3224, in _plan_from_solve_state
    ik_result = self._solve_ik_from_solve_state(
  File "/home/avery/.local/share/ov/pkg/isaac-sim-2023.1.1/kit/python/lib/python3.10/contextlib.py", line 79, in inner
    return func(*args, **kwds)
  File "/home/avery/Documents/curobo/src/curobo/wrap/reacher/motion_gen.py", line 2738, in _solve_ik_from_solve_state
    ik_result = self.ik_solver.solve_any(
  File "/home/avery/Documents/curobo/src/curobo/wrap/reacher/ik_solver.py", line 1161, in solve_any
    return self.solve_single(
  File "/home/avery/Documents/curobo/src/curobo/wrap/reacher/ik_solver.py", line 656, in solve_single
    return self._solve_from_solve_state(
  File "/home/avery/Documents/curobo/src/curobo/wrap/reacher/ik_solver.py", line 1045, in _solve_from_solve_state
    result = self.solver.solve(goal_buffer, coord_position_seed)
  File "/home/avery/Documents/curobo/src/curobo/wrap/wrap_base.py", line 160, in solve
    metrics = self.get_metrics(
  File "/home/avery/Documents/curobo/src/curobo/wrap/wrap_base.py", line 66, in get_metrics
    return self.safety_rollout.get_metrics_cuda_graph(state)
  File "/home/avery/Documents/curobo/src/curobo/rollout/arm_base.py", line 459, in get_metrics_cuda_graph
    self._cu_out_metrics = self.get_metrics(self._cu_metrics_state_in)
  File "/home/avery/Documents/curobo/src/curobo/rollout/arm_base.py", line 436, in get_metrics
    out_metrics = self.convergence_fn(state, out_metrics)
  File "/home/avery/Documents/curobo/src/curobo/rollout/arm_reacher.py", line 401, in convergence_fn
    out_metrics.null_space_error = self.null_convergence.forward_target_idx(
  File "/home/avery/Documents/curobo/src/curobo/rollout/cost/dist_cost.py", line 447, in forward_target_idx
    cost = L2DistLoopFunction.apply(
  File "/home/avery/.local/share/ov/pkg/isaac-sim-2023.1.1/extscache/omni.pip.torch-2_0_1-2.0.2+105.1.lx64/torch-2-0-1/torch/autograd/function.py", line 506, in apply
    return super().apply(*args, **kwargs)  # type: ignore[misc]
  File "/home/avery/Documents/curobo/src/curobo/rollout/cost/dist_cost.py", line 303, in forward
    wp.launch(
  File "/home/avery/.local/share/ov/pkg/isaac-sim-2023.1.1/extscache/omni.warp.core-1.0.0-beta.2/warp/context.py", line 3406, in launch
    raise RuntimeError(
RuntimeError: Failed to find forward kernel 'forward_l2_loop12' from module 'curobo.rollout.cost.dist_cost' for device 'cuda:0'

The same kind of error occurred when running simple_stacking.py

RuntimeError: Failed to find forward kernel 'forward_bound_pos_loop_warp_7' from module 'curobo.rollout.cost.bound_cost' for device 'cuda:0'

[GuoPingPan]

isaac-sim-2023.1.0-hotfix.1, same error when running examples/isaac_sim/ik_reachability.py and examples/isaac_sim/motion_gen_reacher.py using my own robot It is interesting that using franka.yml or examples/ik_example.py will not achieve this error

[GuoPingPan]

@balakumar-s please help.

[balakumar-s]

I am able to reproduce your error, working on a fix.

[balakumar-s]

We just pushed a fix to latest main. Can you confirm and then close this issue?

[curieuxjy]

@balakumar-s Now it works well! Thanks!