Interactive Scene Reconstruction

• [New!] Mapping evaluations added

This repo implements a machine perception pipeline that reconstructs an interactive indoor scene from RGBD streams, where objects are replaced by (articulated) CAD models. Represented as a contact graph, the reconstructed scene naturally encodes actionable information in terms of environmental kinematics, and can be imported into various simulators (via URDF) to support robot interactions.

The pipeline consists of 3 modules:

• A robust panoptic mapping module that accurately reconstruct the semantics and geometry of objects and layouts, which is a modified version of Voxblox++ but with improved robustness. The 2D image segmentation is obtained using Detectron2
• An object-based reasoning module that constructs a contact graph from the dense panoptic map and replaces objects with aligned CAD models
• An interface that converts a contact graph into a kinematic tree in the URDF format, which can be imported into ROS-based simulators

Installation and Usages

• Installation on Ubuntu
• Robust Panoptic Mapping
• [New!] Mapping Evaluations
• Contact Graph and CAD Replacement
• Interactive Scene Generation and Gazebo Example

Citing

• Muzhi Han*, Zeyu Zhang*, Ziyuan Jiao, Xu Xie, Yixin Zhu, Song-Chun Zhu, and Hangxin Liu. Scene Reconstruction with Functional Objects for Robot Autonomy, International Journal of Computer Vision (IJCV), 2022 [Web] [Paper]

• Muzhi Han*, Zeyu Zhang*, Ziyuan Jiao, Xu Xie, Yixin Zhu, Song-Chun Zhu, and Hangxin Liu. Reconstructing Interactive 3D Scenes by Panoptic Mapping and CAD Model Alignments, IEEE International Conference on Robotics and Automation (ICRA), 2021 [Paper] [Arxiv]

@article{han2022scene,
  title={Scene Reconstruction with Functional Objects for Robot Autonomy},
  author={Han, Muzhi and Zhang, Zeyu and Jiao, Ziyuan and Xie, Xu and Zhu, Yixin and Zhu, Song-Chun and Liu, Hangxin},
  journal={International Journal of Computer Vision (IJCV)},
  year={2022},
  publisher={Springer}
}

@inproceedings{han2021reconstructing,
  title={Reconstructing Interactive 3D Scenes by Panoptic Mapping and CAD Model Alignments},
  author={Han, Muzhi and Zhang, Zeyu and Jiao, Ziyuan and Xie, Xu and Zhu, Yixin and Zhu, Song-Chun and Liu, Hangxin},
  booktitle={IEEE International Conference on Robotics and Automation (ICRA)},
  year={2021}
}

Extended Part-based Work

• Zeyu Zhang*, Lexing Zhang*, Zaijin Wang, Ziyuan Jiao, Muzhi Han, Yixin Zhu, Song-Chun Zhu, and Hangxin Liu. Part-level Scene Reconstruction Affords Robot Interaction, IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2023 [Web] [Paper] [Arxiv]

Related Publications in Robot Planning

• Ziyuan Jiao, Yida Niu, Zeyu Zhang, Song-Chun Zhu, Yixin Zhu, and Hangxin Liu. Sequential Manipulation Planning on Scene Graph, IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2022 [web]
• Ziyuan Jiao*, Zeyu Zhang*, Weiqi Wang, David Han, Song-Chun Zhu, Yixin Zhu, and Hangxin Liu. Efficient Task Planning for Mobile Manipulation: a Virtual Kinematic Chain Perspective, IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2021 [web]
• Ziyuan Jiao*, Zeyu Zhang*, Xin Jiang, David Han, Song-Chun Zhu, Yixin Zhu, and Hangxin Liu. Consolidating Kinematic Models to Promote Coordinated Mobile Manipulations, IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2021 [web]