Can't reproduce pitts30k results

[YonatanOfek]

Hey,

I've installed the dependencies as specified in requirements.txt, and have downloaded the models as referenced from the readme, and downloaded the pitts30k after contacting Relja, and formatted it using the convinient VPR-datasets-downloader.

I then used eval.py with the following arguments:

python -m eval --resume_fe=finetunedCCT14_pitts30k.torch --resume_hr=finetunedDHE_pitts30k.torch --datasets_folder=./datasets --dataset_name=pitts30k

Only difference is I had to replace row 120 in model/cct/cct.py as follows:

Because the link in row 25 in model/cct/cct.py appears to be broken:

I am getting these results:

(hmgrphy) PS C:\dev\projects\DHE-VPR> python -m eval --resume_fe=finetunedCCT14_pitts30k.torch --resume_hr=finetunedDHE_pitts30k.torch --datasets_folder=./datasets --dataset_name=pitts30k 2024-07-28 13:45:19 python C:\dev\projects\DHE-VPR\eval.py --resume_fe=finetunedCCT14_pitts30k.torch --resume_hr=finetunedDHE_pitts30k.torch --datasets_folder=./datasets --dataset_name=pitts30k 2024-07-28 13:45:19 Arguments: Namespace(brightness=None, cache_refresh_rate=1000, contrast=None, criterion='triplet', dataset_name='pitts30k', datasets_folder='./datasets', device='cuda', efficient_ram_testing=False, epochs_num=1000, exp_name='default', freeze_te=5, horizontal_flip=False, hue=None, infer_batch_size=32, l2='before_pool', lr=1e-05, majority_weight=0.01, margin=0.1, mining='partial', neg_samples_num=1000, negs_num_per_query=2, num_reranked_preds=32, num_workers=8, optim='adam', patience=3, queries_per_epoch=5000, rand_perspective=None, random_resized_crop=None, random_rotation=None, recall_values=[1, 5, 10, 20], resize=[384, 384], resume_fe='finetunedCCT14_pitts30k.torch', resume_hr='finetunedDHE_pitts30k.torch', saturation=None, save_dir='default', seed=0, test_method='hard_resize', train_batch_size=4, train_positives_dist_threshold=10, trunc_te=8, val_positive_dist_threshold=25) 2024-07-28 13:45:19 The outputs are being saved in logs_test/default/2024-07-28_13-45-19 C:\dev\projects\DHE-VPR\dataset_geoloc.py:43: DeprecationWarning: np.float is a deprecated alias for the builtin float. To silence this warning, use float by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use np.float64 here. Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations self.gallery_utms = np.array([(path.split("@")[1], path.split("@")[2]) for path in self.gallery_paths]).astype(np.float) C:\dev\projects\DHE-VPR\dataset_geoloc.py:44: DeprecationWarning: np.float is a deprecated alias for the builtin float. To silence this warning, use float by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use np.float64 here. Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations self.queries_utms = np.array([(path.split("@")[1], path.split("@")[2]) for path in self.queries_paths]).astype(np.float) 2024-07-28 13:45:22 Geoloc test set: < GeolocDataset, pitts30k - #gallery: 10000; #queries: 6816 > this will not modify any behavior and is safe. When replacing np.int, you may wish to use e.g. np.int64 or np.int32 to specify the precision. If you wish to review your current use, check the release note link for additional information. Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations correct_bool_mat = np.zeros((geoloc_dataset.queries_num, max_recall_value), dtype=np.int)

2024-07-28 13:47:28 baseline test: R@1: 1.2, R@5: 4.7, R@10: 8.2, R@100: 39.3 Testing: 100%|██████████████████████████████████████████████████| 6816/6816 [15:17<00:00, 7.42it/s] 2024-07-28 14:02:47 test after re-ranking - R@1: 1.1, R@5: 5.2, R@10: 9.2, R@20: 14.8

Please let me know if I did anything wrong, and how to reproduce correctly. Thank you, And thanks for the great work.

[Lu-Feng]

Sorry for late reply. The replace at row 120 in cct.py does not affect the result. I guess this failure is probably due to the inconsistency of numpy (or torch) versions. Please refer to requirements.txt

[YonatanOfek]

Thanks!

I reviewed requirements.txt and my versioning is actually correct as far as I can tell. This is my pip list output: Package Version

---

certifi 2024.7.4 charset-normalizer 3.3.2 faiss-gpu 1.7.2 filelock 3.15.4 fsspec 2024.6.1 huggingface-hub 0.24.5 idna 3.7 joblib 1.4.2 kornia 0.5.8 numpy 1.22.4 opencv-python 3.4.13.47 packaging 24.1 Pillow 9.4.0 pip 24.0 PyYAML 6.0.1 regex 2024.7.24 requests 2.32.3 scikit-learn 1.3.2 scipy 1.10.1 setuptools 69.5.1 threadpoolctl 3.5.0 timm 0.3.2 tokenizers 0.12.1 torch 1.8.0 torchvision 0.9.0 tqdm 4.48.2 transformers 4.20.1 typing_extensions 4.12.2 urllib3 2.2.2 wheel 0.43.0

And also this is my conda environment list output if maybe it is relevant: Name Version Build Channel _libgcc_mutex 0.1 conda_forge conda-forge _openmp_mutex 4.5 2_gnu conda-forge blas 1.0 mkl conda-forge bzip2 1.0.8 h4bc722e_7 conda-forge ca-certificates 2024.7.4 hbcca054_0 conda-forge certifi 2024.7.4 pypi_0 pypi charset-normalizer 3.3.2 pypi_0 pypi cudatoolkit 11.1.1 hb139c0e_13 conda-forge faiss-gpu 1.7.2 pypi_0 pypi ffmpeg 4.3 hf484d3e_0 pytorch filelock 3.15.4 pypi_0 pypi freetype 2.10.4 h0708190_1 conda-forge fsspec 2024.6.1 pypi_0 pypi giflib 5.2.2 hd590300_0 conda-forge gmp 6.3.0 hac33072_2 conda-forge gnutls 3.6.13 h85f3911_1 conda-forge huggingface-hub 0.24.5 pypi_0 pypi icu 73.2 h59595ed_0 conda-forge idna 3.7 pypi_0 pypi intel-openmp 2023.1.0 hdb19cb5_46306 joblib 1.4.2 pypi_0 pypi jpeg 9b h024ee3a_2 kornia 0.5.8 pypi_0 pypi lame 3.100 h166bdaf_1003 conda-forge lcms2 2.12 h3be6417_0 ld_impl_linux-64 2.38 h1181459_1 libblas 3.9.0 1_h86c2bf4_netlib conda-forge libcblas 3.9.0 6_ha36c22a_netlib conda-forge libffi 3.4.4 h6a678d5_1 libgcc-ng 14.1.0 h77fa898_0 conda-forge libgfortran-ng 14.1.0 h69a702a_0 conda-forge libgfortran5 14.1.0 hc5f4f2c_0 conda-forge libgomp 14.1.0 h77fa898_0 conda-forge libhwloc 2.11.1 default_hecaa2ac_1000 conda-forge libiconv 1.17 hd590300_2 conda-forge liblapack 3.9.0 6_ha36c22a_netlib conda-forge libpng 1.6.37 h21135ba_2 conda-forge libstdcxx-ng 14.1.0 hc0a3c3a_0 conda-forge libtiff 4.2.0 h85742a9_0 libuv 1.48.0 hd590300_0 conda-forge libwebp 1.2.0 h89dd481_0 libwebp-base 1.2.0 h7f98852_2 conda-forge libxml2 2.13.1 hfdd30dd_2 lz4-c 1.9.3 h9c3ff4c_1 conda-forge mkl 2023.1.0 h213fc3f_46344 ncurses 6.4 h6a678d5_0 nettle 3.6 he412f7d_0 conda-forge ninja 1.12.1 h297d8ca_0 conda-forge numpy 1.22.4 pypi_0 pypi opencv-python 3.4.13.47 pypi_0 pypi openh264 2.1.1 h780b84a_0 conda-forge openssl 3.3.1 h4bc722e_2 conda-forge packaging 24.1 pypi_0 pypi pillow 9.4.0 pypi_0 pypi pip 24.0 py38h06a4308_0 python 3.8.19 h955ad1f_0 python_abi 3.8 2_cp38 conda-forge pytorch 1.8.0 py3.8_cuda11.1_cudnn8.0.5_0 pytorch pyyaml 6.0.1 pypi_0 pypi readline 8.2 h5eee18b_0 regex 2024.7.24 pypi_0 pypi requests 2.32.3 pypi_0 pypi scikit-learn 1.3.2 pypi_0 pypi scipy 1.10.1 pypi_0 pypi setuptools 69.5.1 py38h06a4308_0 sqlite 3.45.3 h5eee18b_0 tbb 2021.12.0 h434a139_3 conda-forge threadpoolctl 3.5.0 pypi_0 pypi timm 0.3.2 pypi_0 pypi tk 8.6.14 h39e8969_0 tokenizers 0.12.1 pypi_0 pypi torchvision 0.9.0 py38_cu111 pytorch tqdm 4.48.2 pypi_0 pypi transformers 4.20.1 pypi_0 pypi typing_extensions 4.12.2 pyha770c72_0 conda-forge urllib3 2.2.2 pypi_0 pypi wheel 0.43.0 py38h06a4308_0 xz 5.4.6 h5eee18b_1 zlib 1.2.13 h5eee18b_1 zstd 1.4.9 ha95c52a_0 conda-forge Maybe cuda 1.11 is bad? Or I should use python version different from 3.8.19?

Thanks

[YonatanOfek]

Actually - maybe it's the scipy and scikit-learn versions? eval.py requires them but I couldn't find which version to install in the requirements.txt, so I installed: scikit-learn 1.3.2 pypi_0 pypi scipy 1.10.1 pypi_0 pypi

[Lu-Feng]

Actually - maybe it's the scipy and scikit-learn versions? eval.py requires them but I couldn't find which version to install in the requirements.txt, so I installed: scikit-learn 1.3.2 pypi_0 pypi scipy 1.10.1 pypi_0 pypi

The version I installed is: scikit-learn==0.24.1 scipy==1.6.2 You can try it.

[YonatanOfek]

Ok - installed these and still doesn't work. Please Let me know what I should try next.

By the way - training (initialization) of msls backbone checkpoint on the pitts 30k dataset starts at ~50 and is at 22.62 after 10 epochs, I don't know if that's expected or not.

Thanks

[Lu-Feng]

It looks like the finetuned model may not load successfully as you run eval.py on Pitts30k. Have you tried evaluation on MSLS using

python eval.py  --resume_fe=/path/to/your/finetunedCCT14_msls.torch --resume_hr=/path/to/your/finetunedDHE_msls.torch --datasets_folder=/path/to/your/datasets_vg/datasets --dataset_name=msls

By the way, why do you train the MSLS backbone on Pitts30k? If everything goes well, the results should not be reduced after the training.

[YonatanOfek]

That's a typo I actually trained the Pitts30k backbone.

I'd like to try using MSLS but I couldn't get the dataset in the right format - VPR-datasets-downloader/format_mapillary.py doesn't seem to make the correct format for eval.py

How should I format MSLS? Thanks

[Lu-Feng]

Running VPR-datasets-downloader/format_mapillary.py directly can get the correct format. I'm not sure what happened in your program. You can also use the model finetuned on MSLS to the test on other datasets, such as SPED and St Lucia, to confirm if there is a problem in the previous model or dataset.

[YonatanOfek]

You're correct format_mapillary.py has no problems, I didn't understand it - Thanks.

MSLS works for eval! I get:

2024-08-03 12:59:48 baseline test: R@1: 74.8, R@5: 87.5, R@10: 90.3, R@100: 96.2 Testing: 100%|████████████████████████████████████████████████| 11084/11084 [24:48<00:00, 7.45it/s] 2024-08-03 13:24:38 test after re-ranking - R@1: 80.2, R@5: 89.1, R@10: 91.5, R@20: 93.1

Although for some reason the Helsinki images were missing from the zip file I downloaded...

Thanks a lot! I might have made some mistake when using Pitts30k, will let you know if I find out what was the problem.

[Lu-Feng]

OK. The difference on MSLS between your results and our paper is because we used the official version of MSLS-val, which only contains 740 query images (a subset of that you used). You can refer to this issue to get results consistent with our paper.