How to implement the ReKep using other robots in the Omnigibson

[goodrain553]

There are quite a lot code assert the robot model is a Fetch. And the IK solver is designed for Fetch. But I want to use other robots like FrankaPanda, What should I do? What part of the code should I modify?

[catslashbin]

I've managed to port most of the setup to FrankaPanda. However, the gripper is still not functioning. While it can plan its movements well without grasping, it currently cannot detect if an object is in its grip.

The process can be tedious, here are some key points that may help.

1. Modify the robot type in config.yml. Also remove the robot.robot_config.controller_config.base as Franka does not have a moving base. You might also change the robot position and the workspace bounds since the robot size differs.
2. Change the action space in environment.py. Fetch uses an action space defined as [lin_vel, ang_vel, dx, dy, dz, dax, day, daz, gripper_l, gripper_r] while Franka uses [dx, dy, dz, dax, day, daz, gripper].
3. Adapt the end-effector coordinates. Franka’s end-effector frame origin is centered on its hand, while Fetch’s origin is positioned between its two gripper fingers. You may need to introduce an offset and apply a rotation in ReKepOGEnv.get_ee_pose().

The IK solver works fine without any changes.

[goodrain553]

I've managed to port most of the setup to FrankaPanda. However, the gripper is still not functioning. While it can plan its movements well without grasping, it currently cannot detect if an object is in its grip.

The process can be tedious, here are some key points that may help.

1. Modify the robot type in config.yml. Also remove the robot.robot_config.controller_config.base as Franka does not have a moving base. You might also change the robot position and the workspace bounds since the robot size differs.
2. Change the action space in environment.py. Fetch uses an action space defined as [lin_vel, ang_vel, dx, dy, dz, dax, day, daz, gripper_l, gripper_r] while Franka uses [dx, dy, dz, dax, day, daz, gripper].
3. Adapt the end-effector coordinates. Franka’s end-effector frame origin is centered on its hand, while Fetch’s origin is positioned between its two gripper fingers. You may need to introduce an offset and apply a rotation in ReKepOGEnv.get_ee_pose().

The IK solver works fine without any changes.

Thanks for your help, I will try that out

[hanlanqian]

I've managed to port most of the setup to FrankaPanda. However, the gripper is still not functioning. While it can plan its movements well without grasping, it currently cannot detect if an object is in its grip. The process can be tedious, here are some key points that may help.

1. Modify the robot type in config.yml. Also remove the robot.robot_config.controller_config.base as Franka does not have a moving base. You might also change the robot position and the workspace bounds since the robot size differs.
2. Change the action space in environment.py. Fetch uses an action space defined as [lin_vel, ang_vel, dx, dy, dz, dax, day, daz, gripper_l, gripper_r] while Franka uses [dx, dy, dz, dax, day, daz, gripper].
3. Adapt the end-effector coordinates. Franka’s end-effector frame origin is centered on its hand, while Fetch’s origin is positioned between its two gripper fingers. You may need to introduce an offset and apply a rotation in ReKepOGEnv.get_ee_pose().

The IK solver works fine without any changes.

Thanks for your help, I will try that out Hello, I'm also endeavoring to replace the Fetch with Panda robot in the demo, but it seems to be failing in the path-solver section (as shown in the following picture). Have you ever encountered similar issues? Thank you in advance!

[catslashbin]

@hanlanqian The orientation of Franka's end-effector frame differs from the Fetch's. You might change the following line to grasp_cost = -np.dot(opt_pose_homo[:3, 2], preferred_dir) + 1 # [0, 1] https://github.com/huangwl18/ReKep/blob/7ec6480829354e39feeafb160393841250261f5f/subgoal_solver.py#L71

[hanlanqian]

@hanlanqian The orientation of Franka's end-effector frame differs from the Fetch's. You might change the following line to grasp_cost = -np.dot(opt_pose_homo[:3, 2], preferred_dir) + 1 # [0, 1]

https://github.com/huangwl18/ReKep/blob/7ec6480829354e39feeafb160393841250261f5f/subgoal_solver.py#L71

Thank you for your prompt response! I discovered that it only aligns the end-effector's z-axis with the negative z-axis of the world frame. After incorporating an additional y-axis grasp cost, the subgoal solver performs effectively, enabling the generated pose to function well in tasks that require vertical pen grasping.