Download and understand control datasets

[pranavguru]

Understanding the schema and format of the control datasets so as to craft optimal prompts for zero-shot and fine-tuned eval of closed source VLMs on control tasks.

[devjwsong]

TODO

• Identifying what each action value means.
• How to update the environment and get the accumulated rewards during the inference stage?
  • The trajectory should change depending on the model's prediction. (Only the first observation is eventually used?)

[devjwsong]

Action definitions

• DM Lab (Discrete) => Need more investigation...
  • https://github.com/google-deepmind/deepmind-research/blob/master/rl_unplugged/dmlab_r2d2.ipynb
• ALE Atari (Discrete)
  • https://gymnasium.farama.org/environments/atari/complete_list
• Baby AI (Discrete)
  • https://minigrid.farama.org/environments/babyai
• MuJoCo (Continuous)
  • https://gymnasium.farama.org/environments/mujoco
• DM Control Suite (Continuous)
  • https://github.com/google-deepmind/dm_control/tree/main/dm_control/suite
• V-D4RL (Continuous)
  • Extended from DM Control Suite
• Meta-World (Continuous)
  • https://arxiv.org/pdf/1910.10897 (The action space is a 2-tuple consisting of the change in 3D space of the end-effector followed by a normalized torque that the gripper fingers should apply. The actions in this space range between −1 and 1.)
  • https://huggingface.co/datasets/jat-project/jat-dataset/viewer/metaworld-peg-unplug-side
  • https://github.com/Farama-Foundation/Metaworld/blob/08489b46faffe3bfdeaa1287fa108260e74e194d/metaworld/envs/mujoco/sawyer_xyz/sawyer_xyz_env.py#L558
• Procgen (Discrete) => Need more investigation...
  • https://github.com/openai/procgen/blob/5e1dbf341d291eff40d1f9e0c0a0d5003643aebf/procgen/env.py#L155
• Language Table (Continuous)
  • https://github.com/google-research/language-table/blob/ee3104061f35e1261a2e28cc2a29ea84e7156661/language_table/environments/language_table.py#L139
  • https://arxiv.org/pdf/2210.06407 (Actions are 2D delta Cartesian setpoints, from the previous setpoint to the new one.)
• Open X-Embodiment (Continuous)
  • https://docs.google.com/spreadsheets/d/1rPBD77tk60AEIGZrGSODwyyzs5FgCU9Uz3h-3_t2A9g/edit?gid=0#gid=0
    • Might need to investigate each dataset more thoroughly...
• LocoMuJoCo (Continuous)
  • https://loco-mujoco.readthedocs.io/en/latest/source/loco_mujoco.api.html#environments

[devjwsong]

Contact to each author?

• DM Lab (First authors of RL Unplugged)
  • Caglar Gulcehre: https://www.caglarg.com
  • Ziyu Wang: https://www.linkedin.com/in/ziyu-wang-433b4614
  • Alexander Novikov: https://www.linkedin.com/in/alexander-novikov-b0a968a6/?originalSubdomain=uk
  • Tom Le Paine: https://tomlepaine.github.io
• Procgen (First author of Procgen)
  • Karl Cobbe: https://www.linkedin.com/in/kcobbe