search

mihaidusmanu / d2-net

D2-Net: A Trainable CNN for Joint Description and Detection of Local Features
Other
761 stars 163 forks source link

Could you help explain about the numerical results in benchmark vs the paper. #69

Closed GabbySuwichaya closed 3 years ago

GabbySuwichaya commented 3 years ago

Hi, thank you for releasing the d2-net code, your kind support, and keep improving the visual localization evaluation. I have tried the Aachen V1.1 and get a reasonable result. It is really awesome.

However, as I have tried to follow the visual localization evaluation benchmark, I often notice that the number in the benchmark such as D2-Net: Single Scale Night 74.5 / 86.7 / 100.0 is much higher than those referenced in research paper such as in Figure 5 of D2-Net's paper 44.9/66.3/88.8 .

Could please explain how the benchmark evaluation results are related to the reported results in the paper? Initially, I thought that they should be the same due to the same settings.

Also, the graph in Figure 5 looks really good. It would be nice if you would release some of those graphs.

mihaidusmanu commented 3 years ago

The benchmark was updated a few weeks ago with more accurate ground-truth poses for the Aachen Day-Night dataset. The results presented in the D2-Net paper are now outdated. Please compare to the results from the website from now on.

Generating the graphs in Figure 5 requires access to the ground truth camera poses which are not publicly available. There are plans to provide plotting capability on the visuallocalization.net website, but I do not have an ETA for that since I am not involved.

GabbySuwichaya commented 3 years ago

I see. Thank you very much.

GabbySuwichaya commented 3 years ago

Also, could you please also maybe provide the link of D2-Net that you would prefer to refer to your work? It seems that there were a number of versions over there.

mihaidusmanu commented 3 years ago

It depends of the evaluation protocol. For retrieval (i.e., NetVLAD) + localization on the full dataset, I suggest comparing to the following submission: https://www.visuallocalization.net/details/964/. For the Local Feature Benchmark code available at https://github.com/tsattler/visuallocalizationbenchmark/tree/master/local_feature_evaluation with predefined retrieved images, you should compare to https://www.visuallocalization.net/details/416/.

At the moment, there are no results on the Aachen v1.1 dataset.

GabbySuwichaya commented 3 years ago

Thanks so much for the support. You have done so many great works. I will keep following :)