search

YanjieZe / 3D-Diffusion-Policy

[RSS 2024] 3D Diffusion Policy: Generalizable Visuomotor Policy Learning via Simple 3D Representations
https://3d-diffusion-policy.github.io
MIT License
552 stars 52 forks source link

Question about reproducing the DP3 algorithm in real-device environment #89

Closed HarveyYesan closed 1 week ago

HarveyYesan commented 1 week ago

We would like to collect data on our own real-device environment to reproduce the DP3 algorithm. We are using a stationary L515 camera, with the body being a RealMan robotic arm and gripper. During the training after data collection, we found that the loss barely decreases. We would like to ask about the following questions: (1) The pointcloud data input into the network is in the camera coordinate system, while the end effector pose in the actions is in the base coordinate system of the robotic arm. Can this be learned? (2) Sometimes the end effector of the robotic arm is outside the camera's visible range. Does this have a significant impact? (3) During trajectory collection, the robotic arm sometimes obscures the target object. Will this have an impact?

YanjieZe commented 1 week ago

Hi, thank you for your interest.

  1. It is ok.
  2. It might hurt performance. It is better to make the robot in the view of the camera.
  3. It is ok.

I think it would be pretty weird if the loss does not decrease at all. This is never observed in our any experiment. even when the policy does not work, the loss should decrease. Have you successfully used DP3 in simulation? I would suspect that there are some problems in your data/setup.