search

CVMI-Lab / UHDM

(ECCV2022) This is the official PyTorch implementation of ECCV2022 paper: Towards Efficient and Scale-Robust Ultra-High-Definition Image Demoireing
Apache License 2.0
191 stars 25 forks source link

Cannot reproduce the results on UHDM #16

Closed zyxxmu closed 1 year ago

zyxxmu commented 1 year ago

Hi authors, Thanks for the great work! I'm currently trying to reproduce the results of ESDNet on UHDM. Unfortunately, I only got 21.2 dB PSNR using the training configuration reported in the paper (150 epochs, Bs: 2; Lr: 2e-4; Scheduler: CosineAnnealingWarmRestarts(optimizer, T_0=50, T_mult=1, eta_min=0.000001, last_epoch=-1) provided in your repo; loss: multi_VGGPerceptualLoss(lam=1, lam_p=1) provided in your repo). I would appreciate any suggestions being given.

XinYu-Andy commented 1 year ago

what’s your patch size and and how do you test it?

zyxxmu commented 1 year ago

Patch size 768, Test code the same as https://github.com/CVMI-Lab/UHDM/blob/99771dc6fc6bdee75bbee1cc5346428526c509f9/model/model.py#L55

XinYu-Andy commented 1 year ago

Patch size 768, Test code the same as

https://github.com/CVMI-Lab/UHDM/blob/99771dc6fc6bdee75bbee1cc5346428526c509f9/model/model.py#L55

Can you provide some output images and their individual metrics ? (e.g. images compared in our paper)

zyxxmu commented 1 year ago

Thanks for your reply. The PSNR difference is fixed by replacing my test code for PSNR (skimage.metrics.peak_signal_noise_ratio) with yours https://github.com/CVMI-Lab/UHDM/blob/99771dc6fc6bdee75bbee1cc5346428526c509f9/utils/metric.py#L23 Your evaluation code for PSNR gives 22.12 dB (similar to the paper), but using skimage.metrics.peak_signal_noise_ratio only gives 21.02 dB. I think the difference may mainly come from converting output fp32 tensors to int8 image arrays or not.

zyxxmu commented 1 year ago

Hi authors, I also trained ESDNet on FHDMi following the configurations given in this repo. (150 epochs, Bs: 2; Lr: 2e-4; Patch size 512). This time I got 22.8 dB PSNR, which is far behind 24.5 dB reported in the paper. I have checked the test code and the evaluation func in your repo is also skimage.metrics.peak_signal_noise_ratio on FHDMi. Therefore, I reopen this issue to figure out if there is any mistakes in my implementation. Many thanks.

XinYu-Andy commented 1 year ago

Hi authors, I also trained ESDNet on FHDMi following the configurations given in this repo. (150 epochs, Bs: 2; Lr: 2e-4; Patch size 512). This time I got 22.8 dB PSNR, which is far behind 24.5 dB reported in the paper. I have checked the test code and the evaluation func in your repo is also skimage.metrics.peak_signal_noise_ratio on FHDMi. Therefore, I reopen this issue to figure out if there is any mistakes in my implementation. Many thanks.

Hi,

We are rushing for ICCV. I suggest that you first check if the output of the test images is normal and whether it can achieve the same results as in our paper. If there is no problem with this, please try our test code to see if the metrics are consistent. Our test code is not unified for all datasets in order to align with previous methods. However, your PSNR is quite lower. Did you save the images as JPEG and cause quality loss? Another way I suggest is to use Matlab for testing. Although the numerical values may be slightly different, it should not vary too much. This can be used to verify whether your training is successful.

zyxxmu commented 1 year ago

Thanks for the suggestion. I‘ll keep trying to fix this. BTW, best wishes for your work ! : )