search

j96w / DexCap

[RSS 2024] "DexCap: Scalable and Portable Mocap Data Collection System for Dexterous Manipulation" code repository
MIT License
181 stars 22 forks source link

Testing results in simulation #6

Open 0xcuriousapple opened 1 month ago

0xcuriousapple commented 1 month ago

Hey @j96w Awesome paper I am trying to imitate my own choice of tasks I am done with data collection and training, and now I want to emulate it in simulation before trying it out on actual leap hand. Any thoughts on how one can test DexCap results in simulated environment (something like Issac, Gazebo..)?

Thanks

0xcuriousapple commented 1 month ago

Hey,

Just to give you an update, I went ahead with using Webots. I'm done with the Webots and controller setup, and now looking to attach a ROS node for inference.

https://github.com/j96w/DexCap/assets/46760063/7eac9e0a-d354-4295-ad18-2afb63c0dfa1

(ps: this is just normal pick and place, there is no AI or imitation learning involved as of now)

If you have any suggestions, please let me know :)

Thanks

j96w commented 3 days ago

Hi @0xcuriousapple, thanks for sharing these! A simulator is definitely helpful. We also considered it earlier, but since DexCap collects human data in the real world, we would need an additional process to build a "digital twin" in the sim to replay the collected human data. How do you plan to handle the real-to-sim gap? Or are you just looking for a teleoperation setup in the sim? DexCap is designed to collect data without a robot (so it's not directly designed for teleop).