Instructions embeddings for RLBench Tasks

[pawanw17]

Hi, Thanks for the great work!

I wanted to generate custom instruction embeddings for the RL bench tasks, I see that you have provided a script for doing so in calvin that uses the dataset annotations, Can you please share anything similar for RLBench?

[nickgkan]

Hi, thanks for your interest in our work!

It seems useful to add such a script indeed. Until then:

1. You can find the language inside each task class, for example here https://github.com/MohitShridhar/RLBench/blob/peract/rlbench/tasks/basketball_in_hoop.py.

2. We had used this script in the past https://github.com/zhouxian/act3d-chained-diffuser/blob/main/data_preprocessing/preprocess_instructions.py.

Hope that helps!

[pawanw17]

That certainly helps, Thanks! Can you please confirm if these annotations are the same ones used while testing the 3d_diffuser_actor

https://github.com/zhouxian/act3d-chained-diffuser/blob/main/data_preprocessing/annotations.json

if not then can you please share the raw annotations/instructions of 3d_diffuser_actor in RLBench?

[twke18]

Hi,

I believe that annotations.json does not include the 18 PerAct tasks, which will instead be loaded from RLBench library. We will need to test the script before updating to the github repo in few days. If you have run the script and find any problem, please let us know. That would be very helpful!

[twke18]

Hi,

We can confirm that the script works out-of-box. We tried the following command:

> python data_preprocessing/preprocess_instructions.py  --tasks place_cups close_jar insert_onto_square_peg light_bulb_in meat_off_grill open_drawer place_shape_in_shape_sorter place_wine_at_rack_location push_buttons put_groceries_in_cupboard put_item_in_drawer put_money_in_safe reach_and_drag slide_block_to_color_target stack_blocks stack_cups sweep_to_dustpan_of_size turn_tap --output instructions.pkl

Notably, the description of open_drawer in RLBench library of PerAct and Hiveformer is slightly different. The embeddings we provided are obtained with Hiveformer's RLBench, since during then, we didn't notice the difference between RLBench libraries.

[pawanw17]

Hi @twke18 Thanks for confirming the script. I think I am good to go now, I am closing the issue.

[fengxiuyaun]

“> python data_preprocessing/preprocess_instructions.py --tasks place_cups close_jar insert_onto_square_peg light_bulb_in meat_off_grill open_drawer place_shape_in_shape_sorter place_wine_at_rack_location push_buttons put_groceries_in_cupboard put_item_in_drawer put_money_in_safe reach_and_drag slide_block_to_color_target stack_blocks stack_cups sweep_to_dustpan_of_size turn_tap --output instructions.pkl”

HI, thanks for your great work! I've noticed that the instruction embedding generated by the script above is not quite consistent with the official one. Is there any issue with that? Eg. task ="push_buttons" and variation=0,

[fengxiuyaun]

And eg. The task "push_buttons" has a total of 48 variations, but the instruction.pkl only contains 18.