cshizhe
commented
1 year ago Hi,
Since there is quite some randomness inside the RLBench, the generated data and evaluation episodes can be different across machines even with the same seed. We suggest using our provided singularity image and run experiments for three times to get more stable results.
Just for your information, the results in the single-task setup with our singularity image and datasets are presented below. The average preformance is similar to what reported in the paper:
| pick_and_lift | pick_up_cup | put_knife | put_money_in_safe | push_button | reach_target | slide_block_to_target | stack_wine | take_money | take_umbrella | Avg. | |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Hiveformer reported | 92.20 | 77.10 | 69.70 | 96.20 | 99.60 | 100.00 | 95.40 | 81.90 | 82.10 | 90.10 | 88.43 |
| Results obtained from the repo | 92.64 | 89.76 | 75.76 | 95.66 | 91.25 | 100.00 | 97.55 | 91.92 | 73.48 | 53.20 | 86.12 |
HuiHuiSun
Hi Pierre-Louis,
The idea of this paper is great, I am trying to reproduce the results in your paper. However, I have encountered some problems in reproducing the single-task learning in Table 2 of your paper. I have only trained three single-tasks, they are Push Button, Put Knife, and Take Umbrella, in which the success rate of Push Button is the same as in the paper, but the success rate of Put Knife is 59.6% , and the success rate of Take Umbrella is 33.4%, which is quite different from Table 2. The training parameters and dataset are from the readme.md, I only modified the path and changed the batchsize to 10. Am I missing something?