search

notmahi / dobb-e

Dobb·E: An open-source, general framework for learning household robotic manipulation
https://dobb-e.com
MIT License
580 stars 52 forks source link
behavior-cloning imitation-learning robot-learning robotics

readme

preview

Dobb·E

arXiv License Code Style: Black PyTorch

Project webpage · Documentation (gitbooks) · Paper

Authors: Mahi Shafiullah*, Anant Rai*, Haritheja Etukuru, Yiqian Liu, Ishan Misra, Soumith Chintala, Lerrel Pinto

Open-source repository of the hardware and software components of Dobb·E and the associated paper, On Bringing Robots Home

https://github.com/notmahi/dobb-e/assets/3000253/d332fa42-d351-4ad0-a305-4b6664bd7170

Abstract

 Throughout history, we have successfully integrated various machines into our homes - dishwashers, laundry machines, stand mixers, and robot vacuums are a few of the latest examples. However, these machines excel at performing a single task effectively. The concept of a “generalist machine” in homes - a domestic assistant that can adapt and learn from our needs, all while remaining cost-effective has long been a northstar in robotics that has been steadily pursued for decades. In this work, we initiate a large-scale effort towards this goal by introducing Dobb·E, an affordable yet versatile general-purpose system for learning robotic manipulation within household settings. Dobb·E can learn a new task with only five minutes of a user showing it how to, thanks to a demonstration collection tool (“The Stick”) we built out of cheap parts and iPhones. We use the Stick to collect 13 hours of data in 22 homes of New York City, and train Home Pretrained Representations (HPR). Then, in a novel home environment, with five minutes of demonstrations and fifteen minutes of adapting the HPR model, we show that Dobb·E can reliably solve the task on the Stretch, a mobile robot readily available in the market. Across roughly 30 days of experimentation in homes of New York City and surrounding areas, we test our system in 10 homes, with a total of 109 tasks in different environments, and finally achieve a success rate of 81%. Beyond success percentages, our experiments reveala plethora of unique challenges absent or ignored in lab-robotics, ranging fromeffects of strong shadows, to demonstration quality by non-expert users. With the hope of accelerating research on home robots, and eventually seeing robot butlers in every home, we open-source Dobb·E software stack and models, our data, and our hardware designs.

What's on this repo

Dobb·E is made out of four major components:

  1. A hardware tool, called The Stick, to comfortably collect robotic demonstrations in homes.
  2. A dataset, called Homes of New York (HoNY), with 1.5 million RGB-D frames. collected with the Stick across 22 homes and 216 environments of New York City.
  3. A pretrained lightweight foundational vision model called Home Pretrained Representations (HPR), trained on the HoNY dataset.
  4. Finally, the platform to tie it all together to deploy it in novel homes, where with only five minutes of training data and 15 minutes of fine-tuning HPR, Dobb·E can solve many simple household tasks.

Reflecting this structure, there are four folders in this repo, where:

  1. hardware contains our 3D printable STL files, as well as instructions on how to set up the Stick.
  2. stick-data-collection contains all the necessary software for processing any data you collect on the Stick.
  3. imitation-in-homes contains our code for training a policy on your collected data using our pretrained models, and also the code to pretrain a new model yourself.
  4. robot-server contains the code to be run on the robot to deploy the learned policies.

The primary documentation source is gitbooks at https://docs.dobb-e.com. There are also associated documentations inside each folder's READMEs.

Paper

paper_preview Get it from ArXiv or our website.

Citation

If you find any of our work useful, please cite us!

@article{shafiullah2023bringing,
  title={On bringing robots home},
  author={Shafiullah, Nur Muhammad Mahi and Rai, Anant and Etukuru, Haritheja and Liu, Yiqian and Misra, Ishan and Chintala, Soumith and Pinto, Lerrel},
  journal={arXiv preprint arXiv:2311.16098},
  year={2023}
}