SAM-6D: Segment Anything Model Meets Zero-Shot 6D Object Pose Estimation

Jiehong Lin, Lihua Liu, Dekun Lu, Kui Jia

CVPR 2024

[Paper]

News

[2024/03/07] We publish an updated version of our paper on ArXiv.
[2024/02/29] Our paper is accepted by CVPR2024!

Update Log

[2024/03/05] We update the demo to support FastSAM, you can do this by specifying SEGMENTOR_MODEL=fastsam in demo.sh.
[2024/03/03] We upload a docker image for running custom data.
[2024/03/01] We update the released model of PEM. For the new model, a larger batchsize of 32 is set, while that of the old is 12.

Overview

In this work, we employ Segment Anything Model as an advanced starting point for zero-shot 6D object pose estimation from RGB-D images, and propose a novel framework, named SAM-6D, which utilizes the following two dedicated sub-networks to realize the focused task:

[x] Instance Segmentation Model
[x] Pose Estimation Model

Getting Started

1. Preparation

Please clone the repository locally:

git clone https://github.com/JiehongLin/SAM-6D.git

Install the environment and download the model checkpoints:

cd SAM-6D
sh prepare.sh

We also provide a docker image for convenience.

2. Evaluation on the custom data

# set the paths
export CAD_PATH=Data/Example/obj_000005.ply    # path to a given cad model(mm)
export RGB_PATH=Data/Example/rgb.png           # path to a given RGB image
export DEPTH_PATH=Data/Example/depth.png       # path to a given depth map(mm)
export CAMERA_PATH=Data/Example/camera.json    # path to given camera intrinsics
export OUTPUT_DIR=Data/Example/outputs         # path to a pre-defined file for saving results

# run inference
cd SAM-6D
sh demo.sh

Citation

If you find our work useful in your research, please consider citing:

@article{lin2023sam,
title={SAM-6D: Segment Anything Model Meets Zero-Shot 6D Object Pose Estimation},
author={Lin, Jiehong and Liu, Lihua and Lu, Dekun and Jia, Kui},
journal={arXiv preprint arXiv:2311.15707},
year={2023}
}