NVlabs / curobo

CUDA Accelerated Robot Library
https://curobo.org
Other
796 stars 125 forks source link

Collapses occur when moving the target cube in the example batch_motion_gen_reacher #219

Closed yanghanjiang closed 4 months ago

yanghanjiang commented 7 months ago
  1. cuRobo version:v0.7.1
  2. cuRobo installation mode:isaac sim
  3. python version:3.10
  4. Isaac Sim version:2023.1.0-hotfix.1
  5. Cuda:11.8
  6. Ubuntu:20.04

Issue Details: Collapses occur when moving the target cube in the example batch_motion_gen_reacher with log below.

2024-04-12 01:57:43 [26,918ms] [Error] [curobo] changing goal type, breaking previous cuda graph.
NoneType: None
Stack (most recent call last):
  File "/home/yhj/verifications/cuRobo/curobo/examples/isaac_sim/batch_motion_gen_reacher.py", line 298, in <module>
    main()
  File "/home/yhj/verifications/cuRobo/curobo/examples/isaac_sim/batch_motion_gen_reacher.py", line 252, in main
    result = motion_gen.plan_batch_env(full_js, ik_goal, plan_config.clone())
  File "/home/yhj/verifications/cuRobo/curobo/src/curobo/wrap/reacher/motion_gen.py", line 1487, in plan_batch_env
    result = self._plan_batch_attempts(
  File "/home/yhj/verifications/cuRobo/curobo/src/curobo/wrap/reacher/motion_gen.py", line 1315, in _plan_batch_attempts
    result = self._plan_from_solve_state_batch(
  File "/home/yhj/verifications/cuRobo/curobo/src/curobo/wrap/reacher/motion_gen.py", line 2173, in _plan_from_solve_state_batch
    traj_result = self._solve_trajopt_from_solve_state(
  File "/home/yhj/Omniverse/Omniverse_Library/isaac_sim-2023.1.0-hotfix.1/kit/python/lib/python3.10/contextlib.py", line 79, in inner
    return func(*args, **kwds)
  File "/home/yhj/verifications/cuRobo/curobo/src/curobo/wrap/reacher/motion_gen.py", line 1050, in _solve_trajopt_from_solve_state
    traj_result = trajopt_instance.solve_any(
  File "/home/yhj/verifications/cuRobo/curobo/src/curobo/wrap/reacher/trajopt.py", line 558, in solve_any
    return self.solve_batch_env(
  File "/home/yhj/verifications/cuRobo/curobo/src/curobo/wrap/reacher/trajopt.py", line 759, in solve_batch_env
    return self.solve_from_solve_state(
  File "/home/yhj/verifications/cuRobo/curobo/src/curobo/wrap/reacher/trajopt.py", line 595, in solve_from_solve_state
    goal_buffer = self.update_goal_buffer(solve_state, goal)
  File "/home/yhj/verifications/cuRobo/curobo/src/curobo/wrap/reacher/trajopt.py", line 497, in update_goal_buffer
    log_error("changing goal type, breaking previous cuda graph.")
  File "/home/yhj/verifications/cuRobo/curobo/src/curobo/util/logger.py", line 44, in log_error
    logger.error(txt, exc_info=exc_info, stack_info=stack_info, *args, **kwargs)
Traceback (most recent call last):
  File "/home/yhj/verifications/cuRobo/curobo/examples/isaac_sim/batch_motion_gen_reacher.py", line 298, in <module>
    main()
  File "/home/yhj/verifications/cuRobo/curobo/examples/isaac_sim/batch_motion_gen_reacher.py", line 252, in main
    result = motion_gen.plan_batch_env(full_js, ik_goal, plan_config.clone())
  File "/home/yhj/verifications/cuRobo/curobo/src/curobo/wrap/reacher/motion_gen.py", line 1487, in plan_batch_env
    result = self._plan_batch_attempts(
  File "/home/yhj/verifications/cuRobo/curobo/src/curobo/wrap/reacher/motion_gen.py", line 1315, in _plan_batch_attempts
    result = self._plan_from_solve_state_batch(
  File "/home/yhj/verifications/cuRobo/curobo/src/curobo/wrap/reacher/motion_gen.py", line 2173, in _plan_from_solve_state_batch
    traj_result = self._solve_trajopt_from_solve_state(
  File "/home/yhj/Omniverse/Omniverse_Library/isaac_sim-2023.1.0-hotfix.1/kit/python/lib/python3.10/contextlib.py", line 79, in inner
    return func(*args, **kwds)
  File "/home/yhj/verifications/cuRobo/curobo/src/curobo/wrap/reacher/motion_gen.py", line 1050, in _solve_trajopt_from_solve_state
    traj_result = trajopt_instance.solve_any(
  File "/home/yhj/verifications/cuRobo/curobo/src/curobo/wrap/reacher/trajopt.py", line 558, in solve_any
    return self.solve_batch_env(
  File "/home/yhj/verifications/cuRobo/curobo/src/curobo/wrap/reacher/trajopt.py", line 759, in solve_batch_env
    return self.solve_from_solve_state(
  File "/home/yhj/verifications/cuRobo/curobo/src/curobo/wrap/reacher/trajopt.py", line 595, in solve_from_solve_state
    goal_buffer = self.update_goal_buffer(solve_state, goal)
  File "/home/yhj/verifications/cuRobo/curobo/src/curobo/wrap/reacher/trajopt.py", line 497, in update_goal_buffer
    log_error("changing goal type, breaking previous cuda graph.")
  File "/home/yhj/verifications/cuRobo/curobo/src/curobo/util/logger.py", line 45, in log_error
    raise
RuntimeError: No active exception to reraise

Found some workarounds, like replacing this line https://github.com/NVlabs/curobo/blob/d6e600c88c9b4fbdad99f8ad3ed6ce514d309c25/examples/isaac_sim/batch_motion_gen_reacher.py#L149 with use_cuda_graph=False,

, but maybe should addressing this issue at its root cause.

balakumar-s commented 7 months ago

We are working on a fix, the alternative is to comment out the next line which is motion_gen.warmup().

balakumar-s commented 4 months ago

Fixed in latest main. Feel free to re-open if issue still persists.