Whether the imagined point cloud of the robot is used in training the diffusion policy?

[v-wewei]

Dear author, thanks for your good work for DP3 policy.

I have just gone through your code for the DP3 encoder, which i think is one of the most important part of your work. I have a problem for whether the imagined point cloud of the robot is used in training. Since this technique is used some other related works for performance boosting. If the imagined point cloud is used, could you please provide the performance comparison for w/o the imagined point cloud.

Thanks advance for your kind help. Looking forward your relply soon!

[YanjieZe]

Hi, thank you!

Imagined point clouds are only used in DexArt, since it is provided originally by the authors. We treat this as part of original visual observations from DexArt env. As you may see, other envs do not have such additional information. I think it could be interesting to see whether removing this imagined point cloud in DexArt would affect the results much.

Notably, DP3 works well without the imagined point cloud in other tasks. We wanted to provide a simple and easy-to-use approach with only visual observations (together with robot states) and you could use DP3 just without the imagined point clouds.