Questions of blocks being thrown out in the inference result of Kuka Stacking.

[mls1999725]

When I run the inference files (conditional_kuka_planning_eval.py and rearrangment_kuka_planning_eval.py) with the pre-trained model, I encounter the problem of blocks being thrown around in the rendering results (as shwon in the videos below).

1. Conditional Stacking Block stacking steps performed by the robot arm is (green->red)->(yellow->green)->(yellow->green)

The Google Drive link for the video is https://drive.google.com/file/d/1tqiJpIEK742NIFnTUOFRwz2oMaH3xL0-/view?usp=sharing.

3. Rearrangement Stacking Block stacking steps performed by the robot arm is (red->green)->(yellow->blue)->(yellow->blue)

The Google Drive link for the video is https://drive.google.com/file/d/154ExrZwEivSRaS7ggkeLou1k6k5qL2nZ/view?usp=sharing

Is the problem of blocks being thrown around and moving violently caused by the fact that the perturbation function for the contact constraint is not added in the code implementation? I found that one difference between the code and the paper is that the perturbation function in the code is the l1 distance of the xy coordinates of the two blocks, as shown in this line: https://github.com/jannerm/diffuser/blob/7c4d4da918c38f30241283d1f70d61a9b2293d09/diffusion/denoising_diffusion_pytorch.py#L454, not the contact constraint described in the paper, as shown in the formula below.

[yilundu]

Hi -- the thrown issue is caused by the model predicting contact formation with a block when it isn't close to the block (leading to a fixed attachment joint being made that causes odd block movement). Usually, it means the scene is now OOD for the learned model.

[mls1999725]

Hi -- the thrown issue is caused by the model predicting contact formation with a block when it isn't close to the block (leading to a fixed attachment joint being made that causes odd block movement). Usually, it means the scene is now OOD for the learned model.

Thanks for your reply.

[mls1999725]

If I add code to the add_fixed_constraint function to check the distance between the pose of end effector and the block as shown below, the constraint will be created only when the distance is less than the threshold. This will cause the end effector outside the threshold to not attach to the block, and the arm will not lift the block, which will cause some steps of the goal to fail to complete. How to solve this situation?

Thank you again.

[yilundu]

You probably need to tune the hyperparameter to determine when to form contact or not