Task distribution for Open-X

[pranavguru]

Make justified decisions on the inclusion of sub-datasets within very large datasets such as OpenX. Are all sub-datasets required, or just a few? If few, which ones and why.

Ref: OpenVLA uses only a subset of OpenX datasets

[pranavguru]

Approach to curate the Open-X Embodiment collection for v0:

• Group the datasets by Robot used to generate the data
• Compare the datasets within each group using the features -['Robot', 'Robot Morphology', 'Gripper', 'Action Space', '# RGB Cams', '# Depth Cams', '# Wrist Cams', 'Language Annotations', 'Has Camera Calibration?', 'Has Proprioception?']
• If any 2 datasets within a group have the same values for all the features mentioned above, choose the dataset with the higher number of episodes and drop the lower one.

Final list of datasets in Open-X for MultiNet v0 after curating :

• RT-1 Robot Action QT-Opt • Berkeley Bridge • Freiburg Franka Play • USC Jaco Play • Berkeley Cable Routing • Roboturk • NYU VINN • Austin VIOLA • Berkeley Autolab UR5 • TOTO Benchmark • Language Table • Columbia PushT Dataset • NYU ROT • Stanford HYDRA • NYU Franka Play • Maniskill • Furniture Bench • CMU Franka Exploration • UCSD Kitchen • UCSD Pick Place • Austin Sirius • BC-Z • USC Cloth Sim • Tokyo PR2 Tabletop Manipulation • Saytap • UTokyo xArm PickPlace • UTokyo xArm Bimanual • Robonet • Berkeley MVP Data • Berkeley RPT Data • KAIST Nonprehensile Objects • QUT Dynamic Grasping • Stanford MaskVIT Data • LSMO Dataset • DLR Sara Pour Dataset • DLR Sara Grid Clamp Dataset • DLR Wheelchair Shared Control • ASU TableTop Manipulation • Stanford Robocook • ETH Agent Affordances • Imperial Wrist Cam • CMU Franka Pick-Insert Data • QUT Dexterous Manpulation • MPI Muscular Proprioception • UIUC D3Field • Austin Mutex • Berkeley Fanuc Manipulation • CMU Food Manipulation • CMU Play Fusion • CMU Stretch • RECON • CoryHall • SACSoN • RoboVQA • ALOHA • DROID • ConqHose • DobbE • FMB • IO-AI Office PicknPlace • MimicPlay • MobileALOHA • RoboSet • TidyBot • VIMA • SPOC • Plex RoboSuite

Final list of datasets in Open-X dropped from MultiNet v0 after curating :

• Austin BUDS
• Austin Sailor
• Stanford Kuka Multimodal
• Tokyo PR2 Fridge Opening

[harshsikka]

@pranavguru we may have new datasets being dropped since this, right? Just wanted to maintain a source of truth

[pranavguru]

Final list used for v0:

RT-1 Robot Action QT-Opt Berkeley Bridge Freiburg Franka Play USC Jaco Play Berkeley Cable Routing Roboturk NYU VINN Austin VIOLA Berkeley Autolab UR5 TOTO Benchmark Language Table Columbia PushT Dataset NYU ROT Stanford HYDRA NYU Franka Play Maniskill Furniture Bench CMU Franka Exploration UCSD Kitchen UCSD Pick Place Austin Sirius BC-Z USC Cloth Sim Tokyo PR2 Fridge Opening Tokyo PR2 Tabletop Manipulation Saytap UTokyo xArm PickPlace UTokyo xArm Bimanual Berkeley MVP Data Berkeley RPT Data KAIST Nonprehensile Objects Stanford MaskVIT Data LSMO Dataset DLR Sara Pour Dataset DLR Sara Grid Clamp Dataset DLR Wheelchair Shared Control ASU TableTop Manipulation Stanford Robocook ETH Agent Affordances Imperial Wrist Cam CMU Franka Pick-Insert Data Austin Mutex CMU Play Fusion CMU Stretch RECON CoryHall SACSoN ConqHose DobbE IO-AI Office PicknPlace RoboSet Plex RoboSuite