wyf2020
commented
8 months ago Hi Sarlin,
Thank you for your interest in our work. We address your questions as follows.
1) Firstly, we clarify that all statistics in Fig. 1 are from Tab. 1, where we evaluate the running time of all methods using unified 640×480 image resolution on ScanNet, as pointed out in Tab. 1 caption and Sec. 4.2 Evaluation protocol. The reason for not using MegaDepth for time evaluation is due to the significantly varied image resolutions used in the baselines’ original papers, as summarized in the following table, where the running times of our method on different MegaDepth resolutions are shown in Tab. 8.
Summary of 13 papers from top conferences; the data in '()' is from the official code, while the rest is from the paper and appendix.
| method | MegaDepth | ScanNet | Ransac/thr | conference |
|---|---|---|---|---|
| SP+SG | 1600 max_keypoints=2048 nms=3 (keypoint_threshold=0.005) | 640×480 max_keypoints=1024 nms=4 (keypoint_threshold=0.005) | OpenCV RANSAC/0.5pix | CVPR2020 |
| LoFTR | 840 | 640×480 | OpenCV RANSAC/0.5pix | CVPR2021 |
| QuadTree | (832) | 640×480 | OpenCV RANSAC/0.5pix | ICLR2022 |
| ASpanFormer | 1152 | 640×480 | OpenCV RANSAC/0.5pix | ECCV2022 |
| MatchFormer | 840 | 640×480 | OpenCV RANSAC/0.5pix | ACCV2022 |
| TopicFM | 1200 | 640×480 | OpenCV RANSAC/0.5pix | AAAI2023 |
| DKM | (880×660) | (640×480) | OpenCV RANSAC/0.5pix | CVPR2023 |
| ASTR | 1216 | 640×480 | OpenCV RANSAC/0.5pix | CVPR2023 |
| PATS | None | (640×480) | OpenCV RANSAC/0.5pix | CVPR2023 |
| CasMTR | 1152 | 640×480 | OpenCV RANSAC/0.5pix | ICCV2023 |
| SP+LG | 1600 max_keypoints=2048 (nms=4 keypoint_threshold=0.0005)[LG] & (nms=3 keypoint_threshold=0)[gluefactory] | Not eval | OpenCV RANSAC or poselib(LO-RANSAC)/self tune | ICCV2023 |
| RoMa | 672 | (560×560) | OpenCV RANSAC/0.5pix | CVPR2024 |
| Ours | 1152 | 640×480 | OpenCV RANSAC/0.5pix | CVPR2024 |
Notably, we can also use ScanNet AUC in Fig. 1’s accuracy comparisons, where the gaps between ours and LG still exist: Compared to the SP+LG AUC (49.9, 67.0, 80.1) on MegaDepth, our AUC (56.4, 72.2, 83.5) increased by (13%, 8%, 4%). Meanwhile, on ScanNet’s generalization results, compared to SP+LG AUC (14.8, 30.8, 47.5), our AUC (19.2, 37.0, 53.6) increased by (30%, 20%, 13%). We show both figures for a comprehensive understanding:
However, we didn't use ScanNet's AUC in Fig. 1 mainly because our experiments found that the quality of MegaDepth is better than ScanNet. (Perhaps this is the reason why LightGlue didn't perform experiments on ScanNet. )
2) As for the accuracy and efficiency comparisons with LightGlue on MegaDepth, given that the strongest model from the LightGlue paper (SP + LG, 1600 input image size with 2048 keypoints, carefully tuned RANSAC thr), the AUC is 49.9, 67.0, 80.1 for AUC@5, 10, 20 respectively and the total running time is SP (46.1ms) + LG (26.8ms) = 72.9ms. Our method with a 640×640 image resolution without changing the RANSAC method and thr (kept same with LoFTR and many other baselines) can already achieve generally better accuracy (51.0, 67.4, 79.8) and faster end-to-end inference speed 41.7ms), as shown in Tab. 8. And our optimized model can even achieve AUC (50.5, 67.1, 79.6)(fixed RANSAC thr) and (51.9, 68.0, 80.0)(tuned RANSAC thr) in just 34.1 ms.
By the way, we kindly remind that the running time of feature extraction & keypoint detection (SuperPoint or DISK) is missing in LightGlue’s Tab. 2, where only matching time (from sparse features input to matches output) is used to compare with dense methods which are end-to-end (from image input to matches output).
3) As for RANSAC setting, we follow the setting used by most of (12/13) the previous methods that use the same fixed RANSAC method (OpenCV) and threshold (0.5px), as summarized above in Table. We also evaluate using LG’s setting that changes RANSAC method and tunes the inlier thresholds. As shown below, the accuracy gaps between ours and LG still exist with more advanced LO-RANSAC.
| RANSAC(follow 12/13 papers)(0.5px) | RANSAC(tuned by LG) | RANSAC(tuned by us)(0.3px) | LO-RANSAC(follow LG in gluefactory)(2.0px) | LO-RANSAC(tuned by us)(1.5px) | |
|---|---|---|---|---|---|
| Ours | 56.4 / 72.2 / 83.5 | None | 58.4 / 73.4 / 84.2 | 69.3 / 80.7 / 88.5 | 69.5 / 80.9 / 88.8 |
| RANSAC(follow 12/13 papers)(0.5px) | RANSAC(tuned by LG)(unknown px) | RANSAC(tuned by us) | LO-RANSAC(tuned by LG in gluefactory)(2.0px) | LO-RANSAC(tuned by us) | |
| SP+LG | None | 49.9 / 67.0 / 80.1 | None | 66.8 / 79.3 / 87.9 | None |
| RANSAC(follow 12/13 papers)(0.5px) | RANSAC(tuned by LG)(unknown px) | RANSAC(tuned by us)(0.3px) | LO-RANSAC(tuned by LG)(unknown px) | LO-RANSAC(tuned by us) | |
| ASpanFormer | 55.3 / 71.5 / 83.1 | 55.3 / 71.5 / 83.1 | 58.3 / 73.3 / 84.2 | 69.4 / 81.1 / 88.9 | None |
Moreover, we observe the AUCs of ASpanFormer and other dense methods in the LightGlue’s Tab.2, RANSAC column, are identical to their original papers (without changing RANSAC thresholds), whereas all methods seem carefully tuned as stated in the LightGlue paper (for example, “tuning the RANSAC threshold yields +7% AUC@5 on SuperGlue” in its sup).
Therefore, we also try to tune the RANSAC threshold (finally set to 0.3 pix same as ours) for AspanFormer, where AUCs go from 55.3 / 71.5 / 83.1 (reported in LightGlue) to 58.3 / 73.3 / 84.2. We think this may reflect that finding the best parameter for each method may require a large tuning range and dense sample steps and may potentially overfit to a specific dataset.
I hope these responses can answer your questions and any discussion is welcome :).
Master-cai
sarlinpe
hit2sjtu
Hi,
Before everyone gets too excited, I need to point out some obvious issues in the evaluation described in the paper.
In Figure 1, the inference time of the semi-dense approaches is largely under-estimated because it is computed at a much lower resolution than the pose accuracy (on MegaDepth). This is evidenced by Table 8: 56.4 AUC@5° and 40.1 ms (Table 1) actually correspond to resolutions 1184×1184 and 640×640, respectively. In reality, the proposed approach is much slower: the inference time at this resolution is 139ms (compare this to LightGlue's 30ms). For the reported inference time, the proposed approach is actually not more accurate than LightGlue (and most likely less).
The same story goes for other semi-dense matchers - for LoFTR it should be much higher than 66ms, closer to 180ms (LightGlue paper, Table 2). Even at this resolution, the accuracy gap might completely vanish when using a modern implementation of RANSAC, as found in PoseLib. Evidence of this can also be found in LightGlue, Table 2 (LO-RANSAC). This can be easily evaluated in glue-factory so this omission is surprising.
We'd appreciate having the authors comment on this - @wyf2020 @hxy-123 @Cuistiano - thank you! cc @Phil26AT