Pretrained model on Clavin does not work well

[chenfengxu714]

Dear authors, congratulations for this great work! I am trying to reproduce the result yet got stuck at evaluation. I used your provided pre-trained model yet I got quite bad performance (almost zero success rate). I think I must miss something. Basically I follow your instruction yet I got the results as below. Could you provide me some hints？

Best regards

0 0 1 0 2 0 3 0 4 0 5 0 6 0 7 0 8 0 9 0 10 0 11 0 12 0 13 1 14 0 15 0 16 0 17 0 18 0 19 1 20 0 21 1 22 0 23 0 24 0 25 0 26 0 27 0 28 0 29 0 30 0 31 0 32 0 33 1 34 1 35 0 36 0 37 0 41 0 38 0 39 0 43 0 40 0

[twke18]

Hi, thanks for your interest!

Would you like to provide more information about your local setup?

For your reference, we provided our training and testing log in this post. Could you confirm that your have set up the CALVIN environment correctly, and use this script for evaluating the released model?

[twke18]

The bad performance could result from the recent update of our model. The default argument of quaternion_format becomes xyzw, however, the CALVIN simulation considers wxyz quaternion. Sorry that we forgot to update training and testing scripts on CALVIN accordingly.

Here are the results of the first 20th instruction chains, which look reasonable:

1 3
0 5
2 5
3 2
4 5
5 2
6 5
7 5
8 1
9 5
10 5
11 2
12 3
13 2
14 5
16 2
15 4
18 0
17 2
20 1
19 2
21 5

We have updated the repo. Hope this update has addressed your issue.

[chenfengxu714]

Nice, it works. Thanks for your suggestion!