PyTorch implementation of the paper:
Rotation-Invariant Transformer for Point Cloud Matching by:
Hao Yu, Zheng Qin, Ji Hou, Mahdi Saleh, Dongsheng Li, Benjamin Busam and Slobodan Ilic.
The intrinsic rotation invariance lies at the core of matching point clouds with handcrafted descriptors. However, it is widely despised by recent deep matchers that obtain the rotation invariance extrinsically via data augmentation. As the finite number of augmented rotations can never span the continuous SO(3) space, these methods usually show instability when facing rotations that are rarely seen. To this end, we introduce RoITr, a Rotation-Invariant Transformer to cope with the pose variations in the point cloud matching task. We contribute both on the local and global levels. Starting from the local level, we introduce an attention mechanism embedded with Point Pair Feature (PPF)-based coordinates to describe the pose-invariant geometry, upon which a novel attention-based encoder-decoder architecture is constructed. We further propose a global transformer with rotation-invariant cross-frame spatial awareness learned by the self-attention mechanism, which significantly improves the feature distinctiveness and makes the model robust with respect to the low overlap. Experiments are conducted on both the rigid and non-rigid public benchmarks, where RoITr outperforms all the state-of-the-art models by a considerable margin in the low-overlapping scenarios. Especially when the rotations are enlarged on the challenging 3DLoMatch benchmark, RoITr surpasses the existing methods by at least 13 and 5 percentage points in terms of Inlier Ratio and Registration Recall, respectively.
Clone the repository:
git clone https://github.com/haoyu94/RoITr.git
cd RoITr
Create conda environment and install requirements:
conda env create -f dependencies.yaml
pip install -r requirements.txt
Compile C++ and CUDA scripts:
cd cpp_wrappers
cd pointops
python setup.py install
cd ..
cd ..
Pretrained model can be downloaded from Google Drive.
Put the downloaded model under ./weights/
.
Please follow CoFiNet for preparing the 3DMatch data and put it under ./data/
.
python main.py configs/train/tdmatch.yaml
Point correspondences are first extracted by running:
python main.py configs/test/tdmatch.yaml
and stored on snapshot/tdmatch_ripoint_transformer_test/3DLoMatch/
.
To evaluate on 3DMatch, please change the benchmark
keyword in configs/test/tdmatch.yaml
from 3DLoMatch
to 3DMatch
.
The evaluation of extracted correspondences and relative poses estimated by RANSAC can be done by running:
sh scripts/benchmark_registration_3dlomatch_c2f.sh
and
sh scripts/benchmark_registration_3dmatch_c2f.sh
for 3DLoMatch and 3DMatch, respectively.
The final results are stored in est_traj/3DMatch/{number of correspondences}/result
.
Change the keyword rotated
in configs/test/tdmatch.yaml
to True
.
Generate correspondences by running:
python main.py configs/test/tdmatch.yaml
Evaluation is done through:
python registration/evaluate_registration_c2f_rotated.py --source_path {path to generated correspondences} --benchmark {3DMatch or 3DLoMatch} --n_points {250, 500, 1000, 2500, 5000}
for example:
python registration/evaluate_registration_c2f_rotated.py --source_path snapshot/tdmatch_ripoint_transformer_test/3DLoMatch --benchmark 3DLoMatch --n_points 2500
Pretrained model can be downloaded from Google Drive.
Put the downloaded model under ./weights/
.
Please follow Lepard for preparing the 4DMatch data and put it under ./data/
.
python main.py configs/train/fdmatch.yaml
Point correspondences are first extracted by running:
python main.py configs/test/fdmatch.yaml
and stored on snapshot/fdmatch_ripoint_transformer_test/4DLoMatch/
.
To evaluate on 4DMatch, please change the split
and benchmark
keywords in configs/test/fdmatch.yaml
to split/4DMatch
and 4DMatch
, respectively.
The evaluation of extracted correspondences can be done by running:
python registration/evaluate_fdmatch.py
The path in line 126 should list all the correspondence files, e.g., ./snapshot/fdmatch_ripoint_transformer_test/4DLoMatch/*.pth
.
We thank the authors for their excellent works!
If you find this repository helpful, please cite:
@inproceedings{yu2023rotation,
title={Rotation-invariant transformer for point cloud matching},
author={Yu, Hao and Qin, Zheng and Hou, Ji and Saleh, Mahdi and Li, Dongsheng and Busam, Benjamin and Ilic, Slobodan},
booktitle={CVPR},
year={2023}
}