octo-models / octo

Octo is a transformer-based robot policy trained on a diverse mix of 800k robot trajectories.
https://octo-models.github.io/
MIT License
899 stars 168 forks source link

How to set stateencoding and actionencoding in another robot (legged robot) #98

Open wuyukun-tong opened 6 months ago

wuyukun-tong commented 6 months ago

Hi I am very confused hoe to modify the following code to fit my own data

`class StateEncoding(IntEnum): """Defines supported proprio state encoding schemes for different datasets."""

NONE = -1  # no state provided
POS_EULER = 1  # EEF XYZ + roll-pitch-yaw + 1 x pad + gripper open/close  
POS_QUAT = 2  # EEF XYZ + quaternion + gripper open/close
JOINT = 3  # 7 x joint angles (padding added if fewer) + gripper open/close
JOINT_BIMANUAL = 4  # 2 x [6 x joint angles + gripper open/close]

class ActionEncoding(IntEnum): """Defines supported action encoding schemes for different datasets."""

EEF_POS = 1  # EEF delta XYZ + roll-pitch-yaw + gripper open/close
JOINT_POS = 2  # 7 x joint delta position + gripper open/close
JOINT_POS_BIMANUAL = 3  # 2 x [6 x joint pos + gripper]`

In our custom dataset the robot state are the imu data and joint position data and the action is velocity of robot.

kpertsch commented 6 months ago

If you want to do finetuning you don't need to actually modify this code, you can just directly modify the finetuning config: https://github.com/octo-models/octo/blob/main/scripts/configs/finetune_config.py

The StateEncoding and ActionEncoding is primarily used to determine the action normalization mask (ie which dimensions should be normalized) here: https://github.com/octo-models/octo/blob/5eaa5c6960398925ae6f52ed072d843d2f3ecb0b/octo/data/oxe/__init__.py#L37

So if you want to use your dataset for OpenX co-training, you can define a new element in the Enums above and then modify the code in the link above to set the normalization mask accordingly (eg for Octo we typically don't normalize the gripper dimension)

wuyukun-tong commented 6 months ago

Thank a lot !

GauthamGaneshPrasad commented 3 weeks ago

Hello, Where do we have to make changes to get joint states as action outputs? Is it in the above mentioned code?