search

Red-Fairy / ZeroShotDayNightDA

[ICCV 2023 oral] Official repository of the paper "Similarity Min-Max: Zero-Shot Day-Night Domain Adaptation"
36 stars 1 forks source link

Can not reproduce result for visual place recognition #6

Closed EvelinYin closed 2 months ago

EvelinYin commented 2 months ago

Hi there. I am trying to train the model for visual place recognition. As in the code, the pretrained daytime model is just borrowed from the pretrained resnet101 for darkening.py. Then for train_night.py, since I didn't find the resume checkpoint specified here, I just disabled loading from the checkpoint, and the code starts training using the pretained resnet101 as default. All other configurations are the default ones specified as in the Readme file (Besides that I have added --sim flag which I think you missed in the instruction).

However, I failed to reproduce the result for this task. In fact, during the 5 training epochs, the reported MAP is decreasing (0.799 down to 0.67) with the decrease of the loss, which is really confusing. Could you help to figure out which part I have missed or is doing wrong? Thanks a lot.

Red-Fairy commented 2 months ago

You may find the post-adaptation model in the updated link. Besides, be aware that the train-time evaluation result is not identical to that of test-time. Please follow the original repo for the correct evaluation practices.

EvelinYin commented 2 months ago

You may find the post-adaptation model in the updated link. Besides, be aware that the train-time evaluation result is not identical to that of test-time. Please follow the original repo for the correct evaluation practices.

Hi thanks so much for your timely reply! I do have followed the evaluation process and have got the reported MAP(90.4%) on the post-adaptation model visual_place_recog_post_adaptation.tar you provided here.

However, when I try to train from scratch by myself (pretrained resnet101 as init for both darkening and train_night ), but I only got MAP to be 83.66%. image

Is it because the missing checkpoint specified in the script here? Should I just simply increase the number of epochs for training in the train_night.py?

Red-Fairy commented 2 months ago

Hi,

Thanks for pointing out the mistake. As mentioned in our paper, our model is adapted from a task-specific model pre-trained on the corresponding daytime dataset, i.e., we first train the model (initialized by resnet-101) using the daytime data only (script train.py), then adapt it to nighttime. I have updated the resume argument to pretrained_model_path to avoid misunderstanding.