Performance gap while reproducing the outdoor evaluation on YFCC

[SuhZhang]

Thank you for releasing the official implementation, especially the evaluation code.

I'm trying to reproduce the pose estimation results in your original paper on YFCC (i.e. 38.72, 59.13, 75.81) with the outdoor model (download with the provided link) by running: ./match_pairs.py --input_dir ~/YFCC100M/yfcc100m --input_pairs assets/yfcc_test_pairs_with_gt_original.txt --output_dir dump_yfcc_test_results --eval --resize 1600 --superglue outdoor --max_keypoints 2048 --nms_radius 3 --resize_float

Uncorrected EXIF rotation data is used as "The SuperGlue paper reports the results of SuperGlue without the corrected rotations". But I got only 38.06, 58.38, 74.67, which seems a big gap from the reported. How can I eliminate this disparity?

[SuhZhang]

System: Ubuntu 18.04 Environment: python 3.6.15 pytorch 1.10.1+cu113 opencv-contrib-python 3.4.11.45 numpy 1.19.5

[sarlinpe]

From the README:

You should get the following table for YFCC (or something very close to it, see this note):

Evaluation Results (mean over 4000 pairs):
AUC@5    AUC@10  AUC@20  Prec    MScore
39.02    59.51   75.72   98.72   23.61

After simplifying the model code and evaluation code and preparing it for release, we made some improvements and tweaks that result in slightly different numbers than what was reported in the paper. The numbers and figures reported in the README were done using Ubuntu 16.04, OpenCV 3.4.5, and PyTorch 1.1.0. Even with matching the library versions, we observed some slight differences across Mac and Ubuntu, which we believe are due to differences in OpenCV's image resize function implementation and randomization of RANSAC.

You are welcome to investigate which parts of the implementation cause the discrepancy.