Reproduction Issue

[kya0a]

Hello, there is a significant discrepancy between my experimental results and the results in the paper. Could you please tell me what might be the reason? Here is my info.log file.

2024-05-13 20:21:19 train.py --backbone ResNet50 --fc_output_dim 2048 --train_dataset_folder /media/a8/Lenovo/processed/train --val_dataset_folder /media/a8/Lenovo/processed/val --test_dataset_folder /home/a8/shr/processed/test 2024-05-13 20:21:19 Arguments: Namespace(M=15, N=3, focal_dist=10, s=100, m=0.4, lambda_lat=1.0, lambda_front=1.0, groups_num=9, min_images_per_class=5, backbone='ResNet50', fc_output_dim=2048, batch_size=32, epochs_num=40, iterations_per_epoch=5000, lr=1e-05, classifiers_lr=0.01, brightness=0.7, contrast=0.7, hue=0.5, saturation=0.7, random_resized_crop=0.5, infer_batch_size=16, positive_dist_threshold=25, resume_train=None, resume_model=None, device='cuda', seed=0, num_workers=8, visualize_classes=0, train_dataset_folder='/media/a8/Lenovo/processed/train', val_dataset_folder='/media/a8/Lenovo/processed/val', test_dataset_folder='/home/a8/shr/processed/test', save_dir='default') 2024-05-13 20:21:19 The outputs are being saved in logs/default/2024-05-13_20-21-19 2024-05-13 20:21:19 Loading pretrained backbone's weights from CosPlace 2024-05-13 20:21:21 There are 1 GPUs and 24 CPUs. 2024-05-13 20:21:21 Using cached dataset cache/sfxl_M15_N3_mipc5.torch 2024-05-13 20:21:23 Using cached dataset cache/sfxl_M15_N3_mipc5.torch 2024-05-13 20:21:56 Using 18 groups 2024-05-13 20:21:56 The 18 groups have respectively the following number of classes [5559, 5559, 5613, 5613, 5580, 5580, 5576, 5576, 5728, 5728, 5692, 5692, 5751, 5751, 5639, 5639, 5768, 5768] 2024-05-13 20:21:56 The 18 groups have respectively the following number of images [617004, 617004, 604080, 604080, 600354, 600354, 612192, 612192, 637020, 637020, 633294, 633294, 642618, 642618, 625716, 625716, 635142, 635142] 2024-05-13 20:21:56 There are 5559 classes for the first group, each epoch has 5000 iterations with batch_size 32, therefore the model sees each class (on average) 28.8 times per epoch 2024-05-13 20:21:56 Validation set: < val - #q: 7983; #db: 8015 > 2024-05-13 20:21:56 Start training ... 2024-05-13 20:42:59 Epoch 00 in 0:21:03, < val - #q: 7983; #db: 8015 >: R@1: 89.7, R@5: 94.3, R@10: 95.6, R@20: 96.6 2024-05-13 21:04:09 Epoch 01 in 0:21:08, < val - #q: 7983; #db: 8015 >: R@1: 91.3, R@5: 95.4, R@10: 96.5, R@20: 97.3 2024-05-13 21:24:27 Epoch 02 in 0:20:17, < val - #q: 7983; #db: 8015 >: R@1: 91.9, R@5: 95.5, R@10: 96.5, R@20: 97.4 2024-05-13 21:44:22 Epoch 03 in 0:19:53, < val - #q: 7983; #db: 8015 >: R@1: 91.7, R@5: 95.3, R@10: 96.4, R@20: 97.2 2024-05-13 22:05:40 Epoch 04 in 0:21:16, < val - #q: 7983; #db: 8015 >: R@1: 92.2, R@5: 95.5, R@10: 96.4, R@20: 97.2 2024-05-13 22:25:11 Epoch 05 in 0:19:29, < val - #q: 7983; #db: 8015 >: R@1: 91.0, R@5: 94.5, R@10: 95.5, R@20: 96.4 2024-05-13 22:45:52 Epoch 06 in 0:20:38, < val - #q: 7983; #db: 8015 >: R@1: 92.3, R@5: 95.7, R@10: 96.5, R@20: 97.3 2024-05-13 23:05:01 Epoch 07 in 0:19:07, < val - #q: 7983; #db: 8015 >: R@1: 92.6, R@5: 95.7, R@10: 96.6, R@20: 97.3 2024-05-13 23:24:07 Epoch 08 in 0:19:03, < val - #q: 7983; #db: 8015 >: R@1: 91.9, R@5: 95.3, R@10: 96.2, R@20: 96.9 2024-05-13 23:44:35 Epoch 09 in 0:20:25, < val - #q: 7983; #db: 8015 >: R@1: 93.1, R@5: 96.3, R@10: 97.1, R@20: 97.7 2024-05-14 00:03:49 Epoch 10 in 0:19:11, < val - #q: 7983; #db: 8015 >: R@1: 92.9, R@5: 96.2, R@10: 97.0, R@20: 97.6 2024-05-14 00:23:17 Epoch 11 in 0:19:25, < val - #q: 7983; #db: 8015 >: R@1: 92.6, R@5: 95.8, R@10: 96.5, R@20: 97.4 2024-05-14 00:42:33 Epoch 12 in 0:19:14, < val - #q: 7983; #db: 8015 >: R@1: 93.1, R@5: 96.1, R@10: 96.9, R@20: 97.5 2024-05-14 01:02:14 Epoch 13 in 0:19:39, < val - #q: 7983; #db: 8015 >: R@1: 92.1, R@5: 95.2, R@10: 96.0, R@20: 96.8 2024-05-14 01:21:32 Epoch 14 in 0:19:15, < val - #q: 7983; #db: 8015 >: R@1: 92.5, R@5: 95.6, R@10: 96.3, R@20: 97.0 2024-05-14 01:40:48 Epoch 15 in 0:19:14, < val - #q: 7983; #db: 8015 >: R@1: 92.6, R@5: 95.6, R@10: 96.4, R@20: 97.1 2024-05-14 01:59:58 Epoch 16 in 0:19:07, < val - #q: 7983; #db: 8015 >: R@1: 92.9, R@5: 95.9, R@10: 96.7, R@20: 97.4 2024-05-14 02:19:55 Epoch 17 in 0:19:55, < val - #q: 7983; #db: 8015 >: R@1: 93.0, R@5: 96.0, R@10: 96.8, R@20: 97.6 2024-05-14 02:39:50 Epoch 18 in 0:19:53, < val - #q: 7983; #db: 8015 >: R@1: 93.3, R@5: 96.2, R@10: 97.1, R@20: 97.7 2024-05-14 02:59:27 Epoch 19 in 0:19:34, < val - #q: 7983; #db: 8015 >: R@1: 93.3, R@5: 96.2, R@10: 97.0, R@20: 97.7 2024-05-14 03:19:08 Epoch 20 in 0:19:39, < val - #q: 7983; #db: 8015 >: R@1: 93.5, R@5: 96.5, R@10: 97.2, R@20: 97.8 2024-05-14 03:38:44 Epoch 21 in 0:19:33, < val - #q: 7983; #db: 8015 >: R@1: 92.7, R@5: 95.8, R@10: 96.6, R@20: 97.3 2024-05-14 03:58:03 Epoch 22 in 0:19:16, < val - #q: 7983; #db: 8015 >: R@1: 93.3, R@5: 96.0, R@10: 96.7, R@20: 97.4 2024-05-14 04:17:26 Epoch 23 in 0:19:20, < val - #q: 7983; #db: 8015 >: R@1: 92.6, R@5: 95.5, R@10: 96.3, R@20: 97.0 2024-05-14 04:36:47 Epoch 24 in 0:19:19, < val - #q: 7983; #db: 8015 >: R@1: 92.9, R@5: 95.7, R@10: 96.6, R@20: 97.2 2024-05-14 04:56:09 Epoch 25 in 0:19:19, < val - #q: 7983; #db: 8015 >: R@1: 93.0, R@5: 95.9, R@10: 96.7, R@20: 97.3 2024-05-14 05:15:37 Epoch 26 in 0:19:26, < val - #q: 7983; #db: 8015 >: R@1: 93.7, R@5: 96.5, R@10: 97.2, R@20: 97.9 2024-05-14 05:35:16 Epoch 27 in 0:19:35, < val - #q: 7983; #db: 8015 >: R@1: 93.5, R@5: 96.6, R@10: 97.3, R@20: 97.8 2024-05-14 05:55:24 Epoch 28 in 0:20:06, < val - #q: 7983; #db: 8015 >: R@1: 93.6, R@5: 96.6, R@10: 97.2, R@20: 97.9 2024-05-14 06:14:41 Epoch 29 in 0:19:14, < val - #q: 7983; #db: 8015 >: R@1: 93.5, R@5: 96.3, R@10: 97.0, R@20: 97.7 2024-05-14 06:33:59 Epoch 30 in 0:19:15, < val - #q: 7983; #db: 8015 >: R@1: 93.5, R@5: 96.5, R@10: 97.1, R@20: 97.7 2024-05-14 06:54:43 Epoch 31 in 0:20:42, < val - #q: 7983; #db: 8015 >: R@1: 93.3, R@5: 96.1, R@10: 96.8, R@20: 97.4 2024-05-14 07:14:21 Epoch 32 in 0:19:36, < val - #q: 7983; #db: 8015 >: R@1: 93.0, R@5: 95.9, R@10: 96.5, R@20: 97.3 2024-05-14 07:33:33 Epoch 33 in 0:19:09, < val - #q: 7983; #db: 8015 >: R@1: 93.5, R@5: 96.4, R@10: 97.2, R@20: 97.7 2024-05-14 07:52:48 Epoch 34 in 0:19:12, < val - #q: 7983; #db: 8015 >: R@1: 93.5, R@5: 96.3, R@10: 97.1, R@20: 97.8 2024-05-14 08:12:11 Epoch 35 in 0:19:21, < val - #q: 7983; #db: 8015 >: R@1: 93.3, R@5: 96.1, R@10: 96.9, R@20: 97.5 2024-05-14 08:32:18 Epoch 36 in 0:20:04, < val - #q: 7983; #db: 8015 >: R@1: 94.3, R@5: 96.8, R@10: 97.5, R@20: 98.0 2024-05-14 08:51:38 Epoch 37 in 0:19:17, < val - #q: 7983; #db: 8015 >: R@1: 94.4, R@5: 96.9, R@10: 97.5, R@20: 98.2 2024-05-14 09:11:38 Epoch 38 in 0:19:57, < val - #q: 7983; #db: 8015 >: R@1: 93.4, R@5: 96.2, R@10: 97.1, R@20: 97.6 2024-05-14 09:33:02 Epoch 39 in 0:21:22, < val - #q: 7983; #db: 8015 >: R@1: 94.4, R@5: 96.8, R@10: 97.4, R@20: 97.8 2024-05-14 09:33:04 Trained for 40 epochs, in total in 13:11:44

2024-05-14 11:40:58 eval.py --backbone ResNet50 --fc_output_dim 2048 --test_dataset_folder /home/a8/shr/processed/test --resume_model /home/a8/shr/EigenPlaces-main/logs/default/2024-05-13_20-21-19/best_model.pth 2024-05-14 11:40:58 Arguments: Namespace(M=15, N=3, focal_dist=10, s=100, m=0.4, lambda_lat=1.0, lambda_front=1.0, groups_num=9, min_images_per_class=5, backbone='ResNet50', fc_output_dim=2048, batch_size=32, epochs_num=40, iterations_per_epoch=5000, lr=1e-05, classifiers_lr=0.01, brightness=0.7, contrast=0.7, hue=0.5, saturation=0.7, random_resized_crop=0.5, infer_batch_size=16, positive_dist_threshold=25, resume_train=None, resume_model='/home/a8/shr/EigenPlaces-main/logs/default/2024-05-13_20-21-19/best_model.pth', device='cuda', seed=0, num_workers=8, visualize_classes=0, train_dataset_folder=None, val_dataset_folder=None, test_dataset_folder='/home/a8/shr/processed/test', save_dir='default') 2024-05-14 11:40:58 The outputs are being saved in logs/default/2024-05-1411-40-58 2024-05-14 11:40:58 There are 1 GPUs and 24 CPUs. 2024-05-14 11:40:58 Loading pretrained backbone's weights from CosPlace 2024-05-14 11:41:35 Loading model from /home/a8/shr/EigenPlaces-main/logs/default/2024-05-13_20-21-19/best_model.pth 2024-05-14 13:34:52 < test - #q: 1000; #db: 2805840 >: R@1: 71.4, R@5: 78.2, R@10: 80.6, R@20: 83.5

[ga1i13o]

Hi, if you test on other datasets, are you able to reproduce our results? try also to test on queries v1 using our pretrained model (you can do so by simply specifying "--resume_model torchhub"

Check also the resolution of the queries. They should have the shorter side of 480 pixels. I think in some older release of the dataset they were not resized

let us know if this solves your issue

[kya0a]

Hi, if you test on other datasets, are you able to reproduce our results? try also to test on queries v1 using our pretrained model (you can do so by simply specifying "--resume_model torchhub"

Check also the resolution of the queries. They should have the shorter side of 480 pixels. I think in some older release of the dataset they were not resized

let us know if this solves your issue

Hello, I have used your pre-trained model to perform tests on V1, and the results are consistent with the paper. The shortest side of the test images is 480 pixels.

[ga1i13o]

I think your issue is due to the fact that you are using the train/processed version of the dataset, which was the downsampled version of the dataset used to train CosPlace. You should pass --train_dataset_folder path/to/sf_xl/raw/train/panoramas, which uses the raw version with all panoramas that will be automatically cropped.