search

QVPR / Patch-NetVLAD

Code for the CVPR2021 paper "Patch-NetVLAD: Multi-Scale Fusion of Locally-Global Descriptors for Place Recognition"
MIT License
525 stars 74 forks source link

reproduce Mapillary val recall #25

Closed bixiaopeng0 closed 3 years ago

bixiaopeng0 commented 3 years ago

Hi, can youj tell me how to reproduce val recall? I try get the val recall by val.py,but the result is bad. global feature with pca(4096) and use weights mapillary_WPCA4096.pth.tar . ====> Recall@1: 0.4649 ====> Recall@5: 0.5378 ====> Recall@10: 0.5581 ====> Recall@20: 0.5770 ====> Recall@50: 0.6054 ====> Recall@100: 0.6162

but in paper the the recall@1(Single-Spatial-Patch-NetVLAD)=77.2 and recall@1(NetVLAD)=60.8.

Could you tell me the some detail about reproduce Mapillary val recall? thanks

StephenHausler commented 3 years ago

Hi @bixiaopeng0, thanks for flagging this issue with us. I suspect we have a bug in val.py, which is causing the low numbers. This will require extensive investigation, we will endeavor to fix the bug as soon as possible.

Tobias-Fischer commented 3 years ago

Hey @StephenHausler - quick reminder re this.

Tobias-Fischer commented 3 years ago

Hiya,

Sorry for taking so long to get back to you, @bixiaopeng0!

The reason why you do not get the same numbers as in the paper is that the https://github.com/QVPR/Patch-NetVLAD/blob/main/patchnetvlad/training_tools/val.py file does not properly calculate the recall values on the MSLS dataset. Instead, please use the official https://github.com/mapillary/mapillary_sls/blob/main/evaluate.py

You can obtain the predictions file for the cph and sf parts of the Mapillary dataset by using https://github.com/QVPR/Patch-NetVLAD/blob/main/feature_match.py

Then concatenate these two files, e.g. using cat:

cat PatchNetVLAD_predictions_cph.txt PatchNetVLAD_predictions_sf.txt > PatchNetVLAD_predictions_combined.txt

Now this file needs to be converted to a format suitable for the evaluate.py provided by the Mapillary authors. I just created a script to do that in https://github.com/QVPR/Patch-NetVLAD/blob/main/patchnetvlad/training_tools/convert_kapture_to_msls.py

Simply call it like so:

python ./convert_kapture_to_msls.py /path/to/PatchNetVLAD_predictions_combined.txt /path/to/PatchNetVLAD_predictions_combined_msls.txt

And then evaluate the performance:

python /path/to/mapillary_sls/evaluate.py --msls-root=/please/add/path --cities=cph,sf --prediction=path/to/PatchNetVLAD_predictions_combined_msls.txt

This leads to R@1=80.5 which is a bit better than what we state in the paper.

Closing here - let us know if you have more questions.