search

tonyzhaozh / act

MIT License
556 stars 141 forks source link

How to Get Human Demonstration Data in Mujoco #17

Open Jessy-Huang opened 5 months ago

Jessy-Huang commented 5 months ago

Hello Tony, I have finished reproducing your open source dataset in a virtual environment, and I want to collect some teaching data in Mujoco simulation environment to train the robot to do other tasks, but there is no code in your open source project, and there is no mention of how to control the movement of robotic arm remotely (keypad or remote control) in the official doc of Mujoco, so I'm not sure how to get human data in Mujoco. Can you provide some help? Thanks Jessy

tonyzhaozh commented 5 months ago

Yes that part of the code is not released. What we did is to use the physical ALOHA leader arms to teleoperate the simulated ALOHA follower arms. The operator watches the rendering of the mujoco env on a monitor.

Check out this function to get a rough sense of it. Note it will not run without ALOHA hardware. https://github.com/tonyzhaozh/act/blob/main/sim_env.py#L246

Jessy-Huang commented 5 months ago

Yes that part of the code is not released. What we did is to use the physical ALOHA leader arms to teleoperate the simulated ALOHA follower arms. The operator watches the rendering of the mujoco env on a monitor.

Check out this function to get a rough sense of it. Note it will not run without ALOHA hardware. https://github.com/tonyzhaozh/act/blob/main/sim_env.py#L246

What is the format of source of data(https://github.com/tonyzhaozh/act/blob/main/sim_env.py#L253，https://github.com/tonyzhaozh/act/blob/main/sim_env.py#L255 )，We have a robotic arm in our lab that also has cameras, and based on your experimental process, we thought we could link the Mujoco emulator to our robotic arm and use the data from our lab's robotic arm to drive the Mujoco emulator to acquire the corresponding data.

Can you post the format of master_bot_left and master_bot_right that you ran out and I'll get the corresponding data on my robotic arm