search

magicleap / SuperGluePretrainedNetwork

SuperGlue: Learning Feature Matching with Graph Neural Networks (CVPR 2020, Oral)
Other
3.31k stars 672 forks source link

Question about test result of my train model on InLoc dataset. #103

Open liyichao108 opened 2 years ago

liyichao108 commented 2 years ago

Hi, Thanks for your amazing work. I have reproduce superglue on megadepth by using the method in the paper, but I have some adoubt about the result. My submission in https://www.visuallocalization.net/ is: aachen1.1 day：89.4 / 96.1 / 98.9 night：78.0 / 90.1 / 99.5 robotCar day： 56.6 / 81.8 / 97.6 night：32.6 / 66.2 / 89.6 InLoc duc1： 45.5 / 63.6 / 77.8 duc2：50.4 / 76.3 / 80.2 And your pretrained model (outdoor weight) result is: aachen1.1 day：89.4/96.2/99.3 night：77.5/90.1/99.5 robotCar day： 57.1 / 81.8 / 97.9 night：34.6 / 69.1 / 90.4 InLoc duc1： 49.5 / 66.26 / 79.3 duc2：58.0 / 78.6 / 82.4 My result on RobotCar is 3% lower than yours, on InLoc is 8% lower than yours, Could you please tell me how to improve the test result on this two datasets?

sarlinpe commented 2 years ago

This looks good, nice job. I don't think that full localization is a good way to compare models and study the differences, as many components can go wrong and have some randomness (3D reconstruction, absolute pose estimation namely). I recommend to first compare models based on relative pose estimation, e.g. on the validation set of the Image Matching Benchmark (although these scenes overlap with our training set).

Regarding InLoc: did your run hloc with temporal consistency? These results do and these ones don't. The image pairs pairs-query-netvlad40.txt correspond to the later.

liyichao108 commented 2 years ago

Thanks for your reply. In our test, we fixed all components of hloc pipeline but superglue model. For example, we load offline features、offline image retrieve result and 3D reconstruction model from same files. We used same parameters in pose estimator for both test. So we think it is comparable.

We want to know if there is any training trick to improve superglue in night or indoor scene with weak texture. Should we need to generate any special train dataset for challenge (night or indoor) scene.

sarlinpe commented 2 years ago

Sorry for the late reply. In the paper, the outdoor model is pretrained on homographies with aggressive photometric augmentation before fine-tuning on MegaDepth. This can help improve the robustness to illumination changes. You could also add this augmentation when training on MegaDepth.