search

QVPR / Patch-NetVLAD

Code for the CVPR2021 paper "Patch-NetVLAD: Multi-Scale Fusion of Locally-Global Descriptors for Place Recognition"
MIT License
510 stars 72 forks source link

Testing pretrained models on Pitts30k #62

Closed UsmanMaqbool closed 1 year ago

UsmanMaqbool commented 1 year ago

The official recalls are 85.1 92.2 94.4 for the NetVLAD. Since, you participated in adding PCA layer to the model and testing on the pitts30K results. Unfortunately, I couldn't achieve the results.

My ENV: I tested the code on Torch 1.12 + Cuda 11.6 and Torch LTS 1.8 with Cuda 11.1. GPU: 3090Ti

Using Your pretrained models

Download: 

wget -O pittsburgh_WPCA4096.pth.tar https://cloudstor.aarnet.edu.au/plus/s/1aoTGbFjsekeKlB/download

Test I updated the resumepath in the performance.ini file and this is how I tested the the netvlad performance.

python feature_extract.py \
  --config_path patchnetvlad/configs/performance.ini \
  --dataset_file_path=pitts30k_imageNames_index.txt \
  --dataset_root_dir=./datasets-place-recognition/Test_Pitts250k/ \
  --output_features_dir=/media/leo/2C737A9872F69ECF/why-so-deepv2-data/pittsburgh/patch-netvlad-features/pitts30k_index \
&& \
python feature_extract.py \
  --config_path patchnetvlad/configs/performance.ini \
  --dataset_file_path=pitts30k_imageNames_query.txt \
  --dataset_root_dir=./datasets-place-recognition/Test_Pitts250k/ \
  --output_features_dir=/media/leo/2C737A9872F69ECF/why-so-deepv2-data/pittsburgh/patch-netvlad-features/pitts30k_query \
&& \
python feature_match.py \
  --config_path patchnetvlad/configs/performance.ini \
  --dataset_root_dir==./datasets-place-recognition/Test_Pitts250k/ \
  --query_file_path=pitts30k_imageNames_query.txt \
  --index_file_path=pitts30k_imageNames_index.txt \
  --query_input_features_dir /media/leo/2C737A9872F69ECF/why-so-deepv2-data/pittsburgh/patch-netvlad-features/pitts30k_query \
  --index_input_features_dir /media/leo/2C737A9872F69ECF/why-so-deepv2-data/pittsburgh/patch-netvlad-features/pitts30k_index \
  --result_save_folder=./patchnetvlad/results/pitts30k-pytorchnetvlad \
  --ground_truth_path=./patchnetvlad/dataset_gt_files/pitts30k_test.npz

Result: Official result of Netvlad on Pitts30K test are 85.1 92.2 94.4, however I could get these result.

====> Recall NetVLAD@1: 0.8367
====> Recall NetVLAD@5: 0.9177
====> Recall NetVLAD@10: 0.9404
====> Recall NetVLAD@20: 0.9598
====> Recall NetVLAD@50: 0.9762
====> Recall NetVLAD@100: 0.9850

Using Nanne's pretrained model and adding PCA

Download: link

Add PCA: I updated cluster size = 64. in the train.ini before run as follows

python add_pca.py \
--config_path patchnetvlad/configs/train.ini \
--resume_path /media/leo/2C737A9872F69ECF/why-so-deepv2-data/pittsburgh/official/nannes/vgg16_netvlad_checkpoint/checkpoints/checkpoint.pth.tar \
--dataset_root_dir=/media/leo/2C737A9872F69ECF/datasets/maqbool-datasets/datasets-place-recognition/Test_Pitts250k \
--dataset_choice=pitts

Result, I run the same process and got test results as follow.

====> Recall NetVLAD@1: 0.8173
====> Recall NetVLAD@5: 0.9062
====> Recall NetVLAD@10: 0.9312
====> Recall NetVLAD@20: 0.9504
====> Recall NetVLAD@50: 0.9676
====> Recall NetVLAD@100: 0.9789

Could you please help / suggest something? I'll be thankful.

Usman

StephenHausler commented 1 year ago

The R@1=85.1 result is using the NetVLAD model in MATLAB, produced by the original NetVLAD paper. To replicate that value you'll need to install NetVLAD on MATLAB using their codebase (https://github.com/Relja/netvlad). To ensure fair benchmarking in our paper, we made sure to only use the codebases provided by the original authors themselves for all benchmarking results, this is the case for AP-GeM, DenseVLAD, NetVLAD, SuperGlue and DELG.

The R@1=83.7 that you are getting with our Pitts network is exactly correct.

In order to make this repository and implementation of NetVLAD a holistic package, we added the functionality to run NetVLAD by itself and re-trained a network on the Pitts30k dataset for this purpose using Pytorch. We noticed that we were unable to attain exactly the same performance as the original NetVLAD, potentially due to the different training architecture (MatConvNet vs PyTorch). However we do perform better than the Nanne version.

UsmanMaqbool commented 1 year ago

Thanks a lot for your response and made me clear. Since I was struggling with difference PyTorch and cuda versions, batch sizes and no. of threads etc., to achieve 85.1 :P