Visual Servoing: ToDOs

[JemuelStanley47]

• [x] Separate threads for visual pose stream and controller
• [x] Transformation between camera frame and hand frame
• [x] Transformation between hand frame and gravity aligned body frame
• [ ] Settle on Object pose convention
• [ ] Switch hand pose frame or object pose frame
• [ ] Settle on get_object_pose() --> Return (pos,rot) / (pos,quat) / (pos,euler)
• [ ] Modify arm move commands' args based on return pose type
• [ ] Fix PD controller
• [ ] Estimate latency

[JemuelStanley47]

Oversight:

1. Achieves only pose, and not orientation (PBVS accounts for only cartesian pose)

   • [ ] Align to object/grasp pose (position, orientation)
   • [ ] Future: Track Delta p_{t}^{t-1} Reference to tracking issue

2. Need to hover with offset along z_axis of grasp/object pose (preferably grasp pose)

   • [ ] Do the math for hovering over object along z-axis of grasp pose
   • [ ] Radius of sphere = offset

3. New objective of visual servoing:

   • [ ] Gripper frame needs to decrease the error between the offset grasp pose on the sphere and current pose
   • [ ] Might need to use the same controller used to decrease the error between robot current pose and robot measured pose
   • [ ] When error is within a specific threshold, decrease z linearly to go to true grasp pose

[JemuelStanley47]

TODOs post discussion (11-March-2024):

• [ ] Get the centroid of object (roboflowv/ lang-prompted)
• [ ] Get waypoints of camera/hand pose based on generated grasp pose (Initial trajectory)
• [ ] Best View Point Planning #50
• • [ ] Objective is to maximize the view of the tool from ray-tracing until a specific threshold(distance between camera and object/grasp pose) is achieved (because further than that)
• • [ ] Modify trajectory based on the planner
• [ ] Perform normal visual-servoing controller

[JemuelStanley47]

Manipulation check with/without VisualServoing

• [ ] Precision and Repeatability check
• • [ ] Without visual servoing
• • [ ] With Visual servoing (Use AprilTag)
• • [ ] Maybe do same experiment with impedance with maximum stiffness?

[JemuelStanley47]

• [ ] XMem Object video segmentation
• • [ ] https://github.com/hkchengrex/XMem
• • [ ] Checkout colab notebook: https://colab.research.google.com/drive/1RXK5QsUo2-CnOiy5AOSjoZggPVHOPh1m?usp=sharing#scrollTo=qYv3kbaQT2w4
• [ ] Segment point-cloud based on this segmentation #49