search

YujiaoShi / cross_view_localization_SAFA

50 stars 8 forks source link

Question on the accuracy of CVUSA dataset #3

Open wtyhub opened 4 years ago

wtyhub commented 4 years ago

Hi @shiyujiao , Thanks for your great job. Recently, I tried to reproduce the results of the CVUSA dataset in your paper according to the guidance of your Github, but the test results obtained by the model I trained and the test results obtained by directly using the model you provided are quite different, and both results fail to meet the results reported in the paper. During training and testing, I only modified the path of the dataset. In order to get the results in the paper, do I need to modify other parameters in the code? The first picture shows the results from I trained model and the second picture shows the results from the model you provided test_cvusa WechatIMG1136 I use one 2080Ti GPU and the version of Tensorflow is 1.14.0

YujiaoShi commented 4 years ago

Hi @shiyujiao , Thanks for your great job. Recently, I tried to reproduce the results of the CVUSA dataset in your paper according to the guidance of your Github, but the test results obtained by the model I trained and the test results obtained by directly using the model you provided are quite different, and both results fail to meet the results reported in the paper. During training and testing, I only modified the path of the dataset. In order to get the results in the paper, do I need to modify other parameters in the code? The first picture shows the results from I trained model and the second picture shows the results from the model you provided test_cvusa WechatIMG1136 I use one 2080Ti GPU and the version of Tensorflow is 1.14.0

Hi,

Thanks for the question.

  1. Regarding your results, please set the training epochs to 100. I just noticed it was set to 10 in this repo.
  2. Regarding the results of the provided model and its difference with our paper. The reported top-1 recall on the CVUSA dataset is 89%, and your tested result is 87%. I think the difference is acceptable and it should be infected by the TensorFlow, NumPy, or Cuda version. I cannot remember exactly the Tensorflow version which I use. It may be 1.10.0 or 1.12.0...and also I use 1080Ti for training, I am not sure whether it will influence the results.

Cheers, Yujiao

YujiaoShi commented 4 years ago

Hi,

I have updated some small errors in the codes.

It works well now under the following system configuration:

OS: Ubuntu 16.04 Cuda: 10.0 Python: 3.7.4 Tensorflow: 1.13.1

Please let me know if there are any other problems.

Thanks, Yujiao

CSLab113 commented 4 years ago

Hi @shiyujiao , Thanks for your great job. Recently, I tried to reproduce the results of the CVUSA dataset in your paper according to the guidance of your Github, but the test results obtained by the model I trained and the test results obtained by directly using the model you provided are quite different, and both results fail to meet the results reported in the paper. During training and testing, I only modified the path of the dataset. In order to get the results in the paper, do I need to modify other parameters in the code? The first picture shows the results from I trained model and the second picture shows the results from the model you provided test_cvusa WechatIMG1136 I use one 2080Ti GPU and the version of Tensorflow is 1.14.0

Hi, I had the same problem. How did you solve it? Thx!

YujiaoShi commented 4 years ago

Hi @shiyujiao , Thanks for your great job. Recently, I tried to reproduce the results of the CVUSA dataset in your paper according to the guidance of your Github, but the test results obtained by the model I trained and the test results obtained by directly using the model you provided are quite different, and both results fail to meet the results reported in the paper. During training and testing, I only modified the path of the dataset. In order to get the results in the paper, do I need to modify other parameters in the code? The first picture shows the results from I trained model and the second picture shows the results from the model you provided test_cvusa WechatIMG1136 I use one 2080Ti GPU and the version of Tensorflow is 1.14.0

Hi, I had the same problem. How did you solve it? Thx!

Have you tried on the uploaded polar transformed images? We recently noticed that using different image-read packages or make a small substitute of the original codes will results in performance drops if you use the trained model. This may indicate the sensitivity of the model, and also reminds me of "attack and defense".

Regarding the training results, please set the training epoch to 100. From my experience, it might because the model has not been fully converged.

Hope this helps.

Yujiao

CSLab113 commented 4 years ago

@shiyujiao Thanks for your help！

Finally, I noticed that loading the VGG16 pre-trained model improved the performance.

YujiaoShi commented 4 years ago

Yes, this is one of the key components. I forgot it...

wudongming97 commented 3 years ago

Yes, this is one of the key components. I forgot it...

Hi, I want to know how to initialize the VGG pre-trained model by myself. When I give up the polar transform, this provided initialized model does work because the graph of TensorFlow is changed. Hope your help!