Quaternion format in evaluation scripts

[pawanw17]

Hi, AFAIK, quaternion format in RLbench is xyzw while in Calvin it is wxyz...

However the evaluation scripts of rlbench the quaternion format is still wxyz while the training scripts it is xyzw https://github.com/nickgkan/3d_diffuser_actor/blob/2ab63276a29464a4b6b223ea5472df6100984db8/online_evaluation_rlbench/eval_peract.sh#L21 https://github.com/nickgkan/3d_diffuser_actor/blob/2ab63276a29464a4b6b223ea5472df6100984db8/online_evaluation_rlbench/eval_gnfactor.sh#L21

and it works somehow... I tried to keep it xyzw with your weights(https://huggingface.co/katefgroup/3d_diffuser_actor/blob/main/diffuser_actor_peract.pth) and it failed

EDIT: Also, I wanted to know if the training script used to generate your weights is train_trajectory_peract.sh or train_keypose_peract.sh

Thanks!

[twke18]

Hi,

Here is our previous discussion (#3 ) about quaternion format. In short, we didn't notice that RLBench uses xyzw quaternions, and trained our model with wxyz quaternions. We ablated these two quaternion formats, and we didn't observe performance difference.

For users who want to try our released model out-of-box, we provide a evaluation script for PerAct and one for GNFactor with wxyz quaternion.

For users who want to train a model with the correct quaternion format, we specify the quaternion format to xyzw.

If you want to reproduce our training setup, you just need to adapt this argument to wxyz.

[pawanw17]

Hi, Thanks for the clarification.

Can you also say about the training script used to generate the diffuser_actor_peract.pth weights?

Is it train_trajectory_peract.sh or train_keypose_peract.sh?

EDIT: Found out it is train_keypose_peract.sh, Is there a script or a way to run the weights generated by train_trajectory_peract.sh?

[nickgkan]

Hi, In that case you need to modify the evaluation script so it predicts trajectories, e.g., https://github.com/nickgkan/3d_diffuser_actor/blob/master/online_evaluation_rlbench/eval_peract.sh#L12.