search

zju3dv / EfficientLoFTR

Apache License 2.0
600 stars 43 forks source link

Issue with ASpanFormer's performance on ScanNet #42

Open LinKInLeLe43 opened 1 week ago

LinKInLeLe43 commented 1 week ago

Hi, thanks for your excellent work!

I evaluated ASpanFormer on ScanNet and achieved significantly higher accuracy: 22.4/41.0/57.9, compared to the 19.6/37.7/54.4 reported in Table 1. Additionally, after shuffling and averaging five results, I still obtained 21.8/40.5/57.3. Since ASpanFormer does not report this cross-domain result and has no open issues on GitHub, I would appreciate any insights you might have.

wyf2020 commented 1 week ago

Thank you for your question! Our cross-domain results for ASpanFormer were obtained by running the indoor script of ASpanFormer, where the ckpt needed to be changed to the outdoor version and the NPE res for training and testing in the indoor config file should be adjusted accordingly. We tested this using the environment provided by ASpanFormer, which includes python=3.8, pytorch=1.8.1 and cudatoolkit=10.2, on a V100 GPU and we performed RANSAC evaluation once.

LinKInLeLe43 commented 4 days ago

Thanks for your reply!

I conducted a quick validation and found that this may be caused by the NPE resolutions as you mentioned. I reported the cross-domain results below for EfficientLoFTR and ASpanFormer using fixed training resolutions of [832, 832] and varied test resolutions.

I am trying to confirm that when the test resolutions of ASpanFormer are set to [480, 640], the results are closer to the metrics reported in your paper. However, EfficientLoFTR sets the test resolutions to [832, 832]. Since using the same test resolutions for ASpanFormer also results in higher accuracy, is there an issue with inconsistent test settings here?

[832,832] [640,640] [480,640]
EfficientLoFTR 18.9/36.6/53.4 18.5/36.2/52.8 18.2/36.0/52.8
ASpanFormer 22.4/41.0/57.9 21.4/39.3/56.2 20.1/38.6/55.4