search

arplaboratory / satellite-thermal-geo-localization

[IROS 2023] Official repository for "Long-range UAV Thermal Geo-localization with Satellite Imagery"
https://xjh19971.github.io/STGL/
MIT License
55 stars 8 forks source link

My reproduction results are low #20

Open wsh506 opened 8 months ago

wsh506 commented 8 months ago

Hi, Thanks for your impressive work. When I tried to reproduce this work, it was difficult to achieve the results reported in the paper. I experimented with the same configuration as the code. This is my training configuration:

2024-02-28 23:36:31   Arguments: Namespace(use_extended_data=True, G_test_norm='batch', G_tanh=False, GAN_epochs_decay=0, GAN_lr_policy='linear', GAN_resize=[512, 512], GAN_mode='lsgan', GAN_upsample='bilinear', GAN_save_freq=0, GAN_norm='batch', G_contrast=True, G_gray=False, G_loss_lambda=100.0, visual_all=False, DA_only_positive=True, D_net='none', G_net='none', lambda_DA=0.1, DA='DANN_after', add_bn=True, remove_relu=False, use_faiss_gpu=True, prior_location_threshold=-1, use_best_n=1, separate_branch=False, weight_decay=0.0, train_batch_size=12, infer_batch_size=48, criterion='triplet', margin=0.1, epochs_num=100, patience=100, lr=0.0001, lr_crn_layer=0.005, lr_crn_net=0.0005, optim='adam', cache_refresh_rate=1000, queries_per_epoch=5000, negs_num_per_query=10, neg_samples_num=1000, mining='partial', backbone='resnet18conv4', l2='before_pool', aggregation='netvlad', netvlad_clusters=64, pca_dim=None, num_non_local=1, non_local=False, channel_bottleneck=128, fc_output_dim=None, conv_output_dim=4096, unfreeze=False, pretrain='imagenet', off_the_shelf='imagenet', trunc_te=None, freeze_te=None, seed=0, resume=None, device='cuda', num_workers=8, resize=[512, 512], test_method='hard_resize', majority_weight=0.01, efficient_ram_testing=False, val_positive_dist_threshold=50, train_positives_dist_threshold=35, recall_values=[1, 5, 10, 20], brightness=None, contrast=None, saturation=None, hue=None, rand_perspective=None, horizontal_flip=False, random_resized_crop=None, random_rotation=None, datasets_folder='datasets', dataset_name='satellite_0_thermalmapping_135_100', pca_dataset_folder=None, save_dir='logs/default/satellite_0_thermalmapping_135_100-2024-02-28_23-36-31-a12ee086-c2a8-4a19-8d6b-3feb96480a18')
2024-02-28 23:36:36   The outputs are being saved in logs/default/satellite_0_thermalmapping_135_100-2024-02-28_23-36-31-a12ee086-c2a8-4a19-8d6b-3feb96480a18
2024-02-28 23:36:36   Using 2 GPUs and 96 CPUs
2024-02-28 23:36:37   Train query set: < TripletsDataset, satellite_0_thermalmapping_135_100 - #database: 10256; #queries: 10256 >
2024-02-28 23:36:46   Extended query set: < TripletsDataset, satellite_0_thermalmapping_135_100 - #database: 79950; #queries: 79950 >
2024-02-28 23:36:48   Val set: < BaseDataset, satellite_0_thermalmapping_135_100 - #database: 13011; #queries: 13011 >
2024-02-28 23:36:51   Test set: < BaseDataset, satellite_0_thermalmapping_135_100 - #database: 26568; #queries: 26568 >
2024-02-28 23:36:58   Domain adapataion: DANN_after
2024-02-28 23:36:58   Output dimension of the model is 4096

This is the GPUs I'm using: 2x3090 24G

This is the evalation resualts:

2024-02-29 18:13:03   Arguments: Namespace(use_extended_data=False, G_test_norm='batch', G_tanh=False, GAN_epochs_decay=0, GAN_lr_policy='linear', GAN_resize=[512, 512], GAN_mode='lsgan', GAN_upsample='bilinear', GAN_save_freq=0, GAN_norm='batch', G_contrast=True, G_gray=False, G_loss_lambda=100.0, visual_all=False, DA_only_positive=False, D_net='none', G_net='none', lambda_DA=1.0, DA='none', add_bn=True, remove_relu=False, use_faiss_gpu=True, prior_location_threshold=-1, use_best_n=1, separate_branch=False, weight_decay=0.0, train_batch_size=4, infer_batch_size=16, criterion='triplet', margin=0.1, epochs_num=1000, patience=3, lr=1e-05, lr_crn_layer=0.005, lr_crn_net=0.0005, optim='adam', cache_refresh_rate=1000, queries_per_epoch=5000, negs_num_per_query=10, neg_samples_num=1000, mining='partial', backbone='resnet18conv4', l2='before_pool', aggregation='netvlad', netvlad_clusters=64, pca_dim=None, num_non_local=1, non_local=False, channel_bottleneck=128, fc_output_dim=None, conv_output_dim=4096, unfreeze=False, pretrain='imagenet', off_the_shelf='imagenet', trunc_te=None, freeze_te=None, seed=0, resume='logs/default/satellite_0_thermalmapping_135_100-2024-02-28_23-36-31-a12ee086-c2a8-4a19-8d6b-3feb96480a18/best_model.pth', device='cuda', num_workers=8, resize=[512, 512], test_method='hard_resize', majority_weight=0.01, efficient_ram_testing=False, val_positive_dist_threshold=50, train_positives_dist_threshold=35, recall_values=[1, 5, 10, 20], brightness=None, contrast=None, saturation=None, hue=None, rand_perspective=None, horizontal_flip=False, random_resized_crop=None, random_rotation=None, datasets_folder='./datasets', dataset_name='satellite_0_thermalmapping_135', pca_dataset_folder=None, save_dir='test/default/satellite_0_thermalmapping_135_100-2024-02-28_23-36-31-a12ee086-c2a8-4a19-8d6b-3feb96480a18/satellite_0_thermalmapping_135-2024-02-29_18-13-03')
2024-02-29 18:13:03   The outputs are being saved in test/default/satellite_0_thermalmapping_135_100-2024-02-28_23-36-31-a12ee086-c2a8-4a19-8d6b-3feb96480a18/satellite_0_thermalmapping_135-2024-02-29_18-13-03
2024-02-29 18:13:05   Resuming model from logs/default/satellite_0_thermalmapping_135_100-2024-02-28_23-36-31-a12ee086-c2a8-4a19-8d6b-3feb96480a18/best_model.pth
2024-02-29 18:13:08   Test set: < BaseDataset, satellite_0_thermalmapping_135 - #database: 26568; #queries: 26568 >
2024-02-29 18:16:18   Final feature dim: 4096
2024-02-29 18:16:21   Recalls on < BaseDataset, satellite_0_thermalmapping_135 - #database: 26568; #queries: 26568 >: R@1: 72.5, R@5: 81.4, R@10: 84.6, R@20: 87.5
2024-02-29 18:16:21   Finished in 0:03:17

Can you help me find where the problem is, or can you provide the best model weights to test for possible problems

xjh19971 commented 8 months ago

Hi, thank you for reading our work. We decided not to release the weights but I am happy to help you replicate it.

First, please check if you pulled the latest version, especially after the commit (https://github.com/arplaboratory/satellite-thermal-geo-localization/commit/84208d3751e4fad3319ba08ed7d9cc97be1d7f40) on Jan 13, which fix a critical bug that can degrade the performance.

Second, I noticed that your train batch size was increased from 4 to 12. I recommend using the batch size of 4 since it leads to more iterations per epoch for better performance. Using a batch size of 12 may require more epochs.

Finally, I have attached the wandb plot for your reference, which shows the training curve. You can compare it with your plots for debugging. Specifically, you should expect that recall1 is ~95 and recall5 is ~98 on the validation set.

image

Please let me know if you need more help with your replication.