search

ethz-asl / hfnet

From Coarse to Fine: Robust Hierarchical Localization at Large Scale with HF-Net (https://arxiv.org/abs/1812.03506)
MIT License
776 stars 185 forks source link

How to do 2D-3D matching by SuperGlue? #47

Closed HawkYJH closed 4 years ago

HawkYJH commented 4 years ago

Thanks for your working first! I noticed Hierarchical+SuperGlue won the first place in the CVPR2020's competition of Long-term localization, and this reponsitory do the 2D-3D mathing by knn+ratio test,and the SuperGlue seems to be used to do 2D-2D mathing.May I ask how to use SuperGlue to do the 2D-3D matching,should I project all the 3D map points to the current frame if I know an approximate 6-DOF pose or project all the 3D map points to the normalized plane if I don't know the 6-DOF pose. And the second question is that erery 3D mappoint stores a distinctive descriptor,and SuperGlue needs a score with the descriptor as input.As for this 3D mappoint,should I choose the score corresponding to this distinctive descriptor or the average score of all frames where this map point is observed? Which could be better? And the last question is that SuperGlue use all the keypoints extracted by superpoint as the input.If I randomly choose 1/10 keypoints in the first image and randomly choose 1/10 keypoints in the second image as the input.Will this affect the effect of superglue? Thank you again for your work and look forward to your reply ！

sarlinpe commented 4 years ago

I just release a toolbox that can perform visual localization with SuperGlue and reproduce the results of the challenge: https://github.com/cvg/Hierarchical-Localization Regarding the first two question: we do 2D-2D matching and lift the database keypoints to 3D using the SfM model. Regarding your last question: randomly dropping keypoints for either of the two images in a pair will of course affect the performance, and might also affect the matching ability of SuperGlue, since it will have fewer unambiguous matches to rely on.