search

dexsuite / dex-retargeting

https://yzqin.github.io/anyteleop/
MIT License
176 stars 22 forks source link

retargetting problem with laptop's built-in camera #31

Open dbdxnuliba opened 1 month ago

dbdxnuliba commented 1 month ago

Very nice work！ And when I used the show_realtime_retargeting.py example with the laptop's built-in camera, the retargeting performed normally when my hand was facing the camera. However, as I gradually rotated my right hand, the retargeting showed a significant deviation. How can we solve this problem? Would using an RGBD camera, like Realsense, resolve it? If so, could you please provide an example code? Thank you very much.

Here is the testingmethod： python3 show_realtime_retargeting.py --robot-name allegro --retargeting-type dexpilot --hand-type right

abc

yzqin commented 1 month ago

Hi @dbdxnuliba

That is an unexpected issue. To further debug this problem, could you help to provide the original RGB video (without mediapipe markers) or image sequences so that I can take a deeper look at the problem you mentioned above?

dbdxnuliba commented 1 month ago

https://github.com/user-attachments/assets/3e77cb76-add3-4ca7-8d7f-11a597e9c29b thanks a lot for your response, and this is the original RGB video record from laptop's built-in camera @yzqin

dbdxnuliba commented 1 month ago

test_result.zip

and this is the test result by run dex-retargeting-0.4.0 python3 show_realtime_retargeting.py --robot-name ability --retargeting-type dexpilot --hand-type right @yzqin ,Please inform me if there are any new developments

yzqin commented 1 month ago

Hi @dbdxnuliba

Sorry I find that I can not open/download your original RGB video record. Can you send me again.

dbdxnuliba commented 4 weeks ago

Hi, @yzqin test.zip

this is the original RGB video record

Hi @dbdxnuliba

Sorry I find that I can not open/download your original RGB video record. Can you send me again.

yzqin commented 4 weeks ago

Dear @dbdxnuliba,

Thank you for sharing this example. I've successfully reproduced the issue on my system. The core problem stems from the poor quality of 3D fingertip position detection in the hand pose estimation.

In the video, even when the thumb and index fingertips are visibly touching, the detected distance between them remains quite large, approximately 0.08 units.

I've made updates to the dex-urdf repository to enhance the fingertip definition for the Allegro hand. While this offers some improvement, the issue persists.

You might want to experiment with the retargeting configuration I've provided below. It includes an increased margin for the finger touching prior. This adjustment helps to some extent, but the results are still unsatisfactory due to the significant detection errors from MediaPipe.

retargeting:
  type: DexPilot
  urdf_path: allegro_hand/allegro_hand_right.urdf

  # Target refers to the retargeting target, which is the robot hand
  wrist_link_name: "wrist"
  finger_tip_link_names: [ "link_15.0_tip", "link_3.0_tip", "link_7.0_tip", "link_11.0_tip" ]
  scaling_factor: 1.6

  # DexPilot parameters
  project_dist: 0.05
  escape_dist: 0.07

  # A smaller alpha means stronger filtering, i.e. more smooth but also larger latency
  low_pass_alpha: 0.2

I will continue investigate this issue. Maybe replace the mediapipe with a better hand pose detector in the example.

dbdxnuliba commented 3 weeks ago

Thank you for your reply. Looking forward to the progress.