search

aysim / comingdowntoearth

Implementation of 'Coming Down to Earth: Satellite-to-Street View Synthesis for Geo-Localization' [CVPR 2021]
MIT License
44 stars 3 forks source link

poor evalulation result #1

Closed qzhai closed 2 years ago

qzhai commented 2 years ago

Hi,

I test the CVUSA benchmark using the released model but get very pool accuracy, Recall(p2s/s2p) tp1:0.20/0.10 tp5:0.63/0.50 tp10:1.15/0.90 tp1%:5.12/4.45 what's wrong with the model?

qzhai commented 2 years ago

Hi,

I test the CVUSA benchmark using the released model but get very pool accuracy, Recall(p2s/s2p) tp1:0.20/0.10 tp5:0.63/0.50 tp10:1.15/0.90 tp1%:5.12/4.45 what's wrong with the model?

Hi,

I test the model on the images after the polar transform, the performance is normal but still worse than the paper. Now I get the result: Recall(p2s/s2p) tp1:84.69/84.73 tp5:94.37/94.41 tp10:96.36/96.26 tp1%:99.05/98.82, the result in the paper is recall1: 92.56, recall5: 97.55, recall10: 98.33, top1%: 99.57

aysim commented 2 years ago

Hey,

Thanks for your comment! I had a look at the data preparation script and realised that there was actually a small discrepancy: The polar transformation was computed on the resolution 224x1232 (double resolution). I updated the code according to that. Please run data/convert_polar.py to apply polar transformation on satellite images. Then run the test script.

However, I have to say that the results only change slightly between these two settings, nowhere near the numbers you mention. Anyways, the code now reproduces the numbers from the paper exactly. Below are the numbers I got:

Screenshot 2022-01-17 at 15 48 01
steven30currry commented 1 year ago

Hello, author. Could you please provide the code of ssim, psnr and sd? I used other codes to test, and found that the effect was poor. In addition, can you upload this best checkpoint, thank you very much, just for academic research

Hey,

Thanks for your comment! I had a look at the data preparation script and realised that there was actually a small discrepancy: The polar transformation was computed on the resolution 224x1232 (double resolution). I updated the code according to that. Please run data/convert_polar.py to apply polar transformation on satellite images. Then run the test script.

However, I have to say that the results only change slightly between these two settings, nowhere near the numbers you mention. Anyways, the code now reproduces the numbers from the paper exactly. Below are the numbers I got: Screenshot 2022-01-17 at 15 48 01

steven30currry commented 1 year ago

Hey,

Thanks for your comment! I had a look at the data preparation script and realised that there was actually a small discrepancy: The polar transformation was computed on the resolution 224x1232 (double resolution). I updated the code according to that. Please run data/convert_polar.py to apply polar transformation on satellite images. Then run the test script.

However, I have to say that the results only change slightly between these two settings, nowhere near the numbers you mention. Anyways, the code now reproduces the numbers from the paper exactly. Below are the numbers I got: Screenshot 2022-01-17 at 15 48 01

image This is the best training metric I've gotten with your code so far, and it's still far from the paper