search

swz30 / MIRNet

[ECCV 2020] Learning Enriched Features for Real Image Restoration and Enhancement. SOTA results for image denoising, super-resolution, and image enhancement.
Other
667 stars 96 forks source link

Maybe the checkpoints of enhancemenet on LOL datasets is wrong? #15

Closed peachis closed 3 years ago

peachis commented 3 years ago

Your work is very productive, but I found that when reproducing the SSIM and PSNR metrics on the LOL dataset, the enhanced results obtained with the checkpoints you provided were different from here's and the scores were a bit worse. I wonder if there is any step I did wrong?

swz30 commented 3 years ago

Hi @peachis

Could you please provide the SSIM/PSNR scores that you are getting with the checkpoints?

peachis commented 3 years ago

Hi @swz30, I got PSNR: 21.2725, SSIM: 0.8165 with code:

from skimage.measure import compare_ssim, compare_psnr ... ssim = compare_ssim(rgb_in, rgb_gt, data_range=255, multichannel =True) psnr = compare_psnr(rgb_in, rgb_gt, data_range=255 ) ...

swz30 commented 3 years ago

@peachis We use MATLAB to compute PSNR/SSIM scores. The SSIM implementation is provided here.

peachis commented 3 years ago

@swz30 Thank you for providing details. The PSNR/SSIM obtained with the images you provided is 24.1011/0.8421 (use skimage.measure), which is similar to the result you mentioned. While the images enhanced with your checkpoint can only get 21.2725/0.8165. I compared the images you provided with the one enhanced with checkpoint and found that they are indeed different. Maybe I missed some steps?

swz30 commented 3 years ago

We just checked on our end, and I confirm that the numbers (24.14 PSNR) are reproducible using pre-trained weights. We checked with both the MATLAB and skimage==0.16.2.

peachis commented 3 years ago

@swz30 Thanks a lot! Finally, I find the difference. I clone your code before long in 2020.08, and the images loading by img = lycon.load(filepath). I'm not familiar with lycon and just change it to cv2.imread(filepath), while the right approach is img = cv2.cvtColor(cv2.imread(filepath), cv2.COLOR_BGR2RGB) as your current version. I got the same scores as your mention now. Thanks again!