Help reproducing cup arranging task

[sirwart]

I'm trying to reproduce the cup arranging task using the pre-trained checkpoint. When I run it my arm starts to go nuts (writhe in all directions).

To debug it, I looked at the outputs of policy for the first observation. Here is the output of the policy that's been passed through get_real_umi_action:

action = 
[[ 0.1914, -0.0103,  0.0675, -0.158 ,  0.4563,  0.0026,  0.0903],
 [ 0.332 , -0.0674,  0.2605, -0.1588,  1.7837,  0.9145,  0.0882],
 [ 0.106 ,  0.0006,  0.3121, -0.7326,  0.67  ,  0.0385,  0.0906],
 [ 0.1506, -0.0715,  0.3616, -0.0239,  0.5017,  0.2986,  0.0906],
 [ 0.5219,  0.031 ,  0.0285, -0.3815,  0.1622, -0.2435,  0.0841],
 [ 0.6206, -0.0691,  0.5133, -0.9307, -0.3093, -0.2785,  0.0885],
 [ 0.4292,  0.0545,  0.2291,  0.0067,  0.9565, -0.3452,  0.0906],
 [ 0.3204,  0.137 ,  0.3674,  0.3146,  0.5091,  0.4602,  0.0906],
 [ 0.4759,  0.015 ,  0.227 ,  0.3432,  0.8916, -0.4273,  0.0897],
 [ 0.2003, -0.1353,  0.1784, -0.1041,  1.6625,  0.6456,  0.0808],
 [ 0.1624,  0.0373,  0.1323,  0.4304,  0.989 , -0.0285,  0.0836],
 [ 0.1347, -0.0015, -0.157 ,  0.3469,  2.1637, -0.4349,  0.0859],
 [ 0.2741, -0.1414,  0.3928,  0.2504,  0.8465,  0.2287,  0.0906],
 [ 0.2959,  0.1048,  0.2066,  0.786 ,  0.3266,  0.3725,  0.0906],
 [ 0.0924,  0.0105,  0.1356,  0.53  ,  0.884 ,  0.0471,  0.0903],
 [ 0.0216,  0.0544,  0.1702,  0.5112,  1.0929, -0.1559,  0.0905]]

If I understand the format of action correctly, it represents (x, y, z, rx, ry, rz, gripper pos) in meters/rads in EE space, since the relative offset has already been applied to the base pose. To me the actions explain why the arm is writhing, since if you interpret the data that way it's a really large change in pose from frame to frame and it doesn't seem to be following much of a trajectory except for the gripper pos. The rest of the values look somewhat random.

As for the observation I'm passing to the policy, I'm giving it:

obs['robot0_eef_pos'] = [
  [0.3079, 0.002 , 0.2051],
  [0.3079, 0.002 , 0.2051]]
obs['robot0_eef_rot_axis_angle'] = [
  [0.0118, 0.8268, 0.0094],
  [0.0118, 0.8268, 0.0094]]
obs['camera0_rgb] = image below

I will keep digging but I was wondering if anything stood out to you as wrong. I'm not using one of the supported arms (ViperX), so there might be an issue with my code downstream of the policy, but since I wasn't able to make sense of the policy output I wanted to check here first in case I'm misunderstanding something.

P.S. thanks for the advice on the IK filtering earlier. I realize now my issue was with how I was using EE positioning with my robot arm

[caijun199702]

Hi, I want to reproduce this work too, Weren't you using the ur5e robotic arm? What configuration parameters did you modify to adapt it to your own robotic arm? I am using Epson robot, but I am wondering what configuration or code I should change to apply to this work...

[sirwart]

@caijun199702 I'm using the ViperX 300 6DoF for the arm. You'll need to add a new interpolation controller for both the arm and gripper you're using (I modelled mine off of franka_interpolation_controller.py, with code adapted to fit my arm), and then construct them in umi/real_world/bimanual_umi_env.py when the proper robot type is passed in via CLI.

[caijun199702]

@caijun199702 I'm using the ViperX 300 6DoF for the arm. You'll need to add a new interpolation controller for both the arm and gripper you're using (I modelled mine off of franka_interpolation_controller.py, with code adapted to fit my arm), and then construct them in umi/real_world/bimanual_umi_env.py when the proper robot type is passed in via CLI.

I also have Viper X 300 6DoF robot which is bought for reproducing mobile aloha, can you share the franka_interpolation_controller.py file for me, I want to try Viper robot on UMI later, thank you very much.

[sirwart]

@caijun199702 I pushed my code changes to a fork that includes viperx_interpolation_controller.py: https://github.com/sirwart/universal_manipulation_interface. You can try that code, but since I haven't gotten it to reproduce there are likely still issues with the code.

You'll also need to print mounts for the grippers and camera/mirror for the viper. I've designed those here: https://cad.onshape.com/documents/ac8fabf30608611a6dab91ac/w/fc2d827681add1f7a93a64e5/e/8ae2dc8a09c4321208e2807d

[caijun199702]

@caijun199702 I pushed my code changes to a fork that includes viperx_interpolation_controller.py: https://github.com/sirwart/universal_manipulation_interface. You can try that code, but since I haven't gotten it to reproduce there are likely still issues with the code.

You'll also need to print mounts for the grippers and camera/mirror for the viper. I've designed those here: https://cad.onshape.com/documents/ac8fabf30608611a6dab91ac/w/fc2d827681add1f7a93a64e5/e/8ae2dc8a09c4321208e2807d

You are amazing, thanks a lot!

[jash101]

@sirwart i'm curious if you're facing any issues with the viperx due to it's low payload capability? The specs say you can only use max payload 750g upto 50% reach

[sirwart]

@jash101 it really depends on what task you’re trying to do. For arranging the espresso cup it’s plenty, but for some other tasks I’m thinking of having it do it’s not enough so I’m looking into getting a more capable arm.

[JinmingM]

@sirwart Hello, Have you successfully reproduct this? I noticed that on the Franka, it need to adjust position and rotation settings, but the UR5 doesn’t require this. Does the Viperx robotic arm need similar settings?

[ErinZhang1998]

@sirwart How did you control the gripper width for ViperX 300 6DoF? I saw that in https://github.com/sirwart/universal_manipulation_interface/blob/30f148e7ac84880cd16b6a3d048160d0c8f12d1f/umi/real_world/viperx_interpolation_controller.py#L94 , the gripper seems to be always at 1.0?

[sirwart]

@ErinZhang1998 I never got the arm code to work so I never bothered getting the gripper to work. It shouldn't be a big change however.

[kafei123456]

@ErinZhang1998 I never got the arm code to work so I never bothered getting the gripper to work. It shouldn't be a big change however.

Hey, I have found out the problem. you can check out your gripper width. The gripper width must be in range of real width(0-0.08mm). I guess you used unreal value of gripper width when eval the model.

[noooob-coder]

Hello, I am also currently trying to reproduce this work and I would like to know how in this work the author converts the EE pose into a 6d movement of the arm.