Reproducing the results on ImageNet-Val

[rv-chittersu]

Hi. To evaluate our setup we are trying to reproduce the results mentioned in the paper. To do so.. we have followed the following steps mentioned in readme.

1. Sample 3k images from the ImageNet-Val set using the script (https://github.com/zsyOAOA/ResShift/blob/master/scripts/prepare_testing_imagenet.py)
2. Generate reconstruction images with inference script and models shared CUDA_VISIBLE_DEVICES=gpu_id python inference_resshift.py -i [image folder/image path] -o [result folder] --scale 4 --task realsrx4 --chop_size 512

The PSNR and SSIM from this set aren't matching the numbers reported in papers. Can you confirm the steps and add if we are missing anything?

[zsyOAOA]

Did you evaluate the PSNR and SSIM metrics on the Y channel of the YCbCr space?

[han265]

Hello, I have the same question. I evaluated the two metrics on the Y channel, and the PSNR result obtained using the code from 'https://github.com/chaofengc/IQA-PyTorch' is 24.87; the PSNR obtained using 'utils/util_image.py' is 24.86. Similarly, the results of SSIM are respectively 0.644 and 0.668. All of them are worse than the results in the paper. So what's wrong? Thank you for your reply ahead of time! @zsyOAOA

[han265]

Besides, the CLIPIQA and MUSIQ from 'https://github.com/chaofengc/IQA-PyTorch' are 0.603 and 53.952 respectively, which are better than the result in the paper. What caused that?

[zsyOAOA]

1. You can access my testing dataset via this link and have a test again.
2. There will be a little difference due to randomness.

[han265]

Yes, you are right! Thank you!

[deepalisingh11]

Did you evaluate the PSNR and SSIM metrics on the Y channel of the YCbCr space?

Hey @zsyOAOA I was curious as to why metrics are to be evaluated only on the Y channel space. Can you briefly explain the reason?

[zsyOAOA]

This is a common setting in super-resolution, since we are more sensitive to the Y Channel. @deepalisingh11

[deepalisingh11]

Great, thanks! Makes sense.