PerAct

Annotated Tutorial

Important: Before starting, change the runtime to GPU.

This notebook is an annotated tutorial on training Perceiver-Actor (PerAct) from scratch. We will look at training a single-task agent on the open_drawer task. The tutorial will start from loading calibrated RGB-D images, and end with visualizing action detections in voxelized observations. Overall, this guide is meant to complement the paper by providing concrete implementation details.

Credit

This notebook heavily builds on data-loading and pre-preprocessing code from ARM, YARR, PyRep, RLBench by Stephen James et al. The PerceiverIO code is adapted from perceiver-pytorch by Phil Wang (lucidrains). The optimizer is based on this LAMB implementation. See the corresponding licenses below.

Licenses

• ARM License
• YARR Licence (Apache 2.0)
• RLBench Licence
• PyRep License (MIT)
• Perceiver PyTorch License (MIT)
• LAMB License (MIT)

Citations

PerAct

@inproceedings{shridhar2022peract,
  title     = {Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulation},
  author    = {Shridhar, Mohit and Manuelli, Lucas and Fox, Dieter},
  booktitle = {Proceedings of the 6th Conference on Robot Learning (CoRL)},
  year      = {2022},
}

C2FARM

@inproceedings{james2022coarse,
  title={Coarse-to-fine q-attention: Efficient learning for visual robotic manipulation via discretisation},
  author={James, Stephen and Wada, Kentaro and Laidlow, Tristan and Davison, Andrew J},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  pages={13739--13748},
  year={2022}
}

PerceiverIO

@article{jaegle2021perceiver,
  title={Perceiver io: A general architecture for structured inputs \& outputs},
  author={Jaegle, Andrew and Borgeaud, Sebastian and Alayrac, Jean-Baptiste and Doersch, Carl and Ionescu, Catalin and Ding, David and Koppula, Skanda and Zoran, Daniel and Brock, Andrew and Shelhamer, Evan and others},
  journal={arXiv preprint arXiv:2107.14795},
  year={2021}
}

RLBench

@article{james2020rlbench,
  title={Rlbench: The robot learning benchmark \& learning environment},
  author={James, Stephen and Ma, Zicong and Arrojo, David Rovick and Davison, Andrew J},
  journal={IEEE Robotics and Automation Letters},
  volume={5},
  number={2},
  pages={3019--3026},
  year={2020},
  publisher={IEEE}
}