PSNR of CUFED on paper is inconsistent with real result

[xiab3369]

python test.py --resume ./pretrained_weights/masa_rec.pth --testset TestSet --name masa_rec_TestSet

[xiab3369]

--------- average PSNR: 27.354309, SSIM: 0.814047 The PSNR/SSIM on paper is 27.54/0.814

[xiab3369]

We use Ref level1 image in CUFED5 as reference as TTSR did.

[SkyeLu]

TestSet only use one reference image. While as mentioned in our paper, we stitch 4 references to one image, same as the setting of TTSR. Please try the following command: python test.py --resume './pretrained_weights/masa_rec.pth' --testset TestSet_multi --name masa_TestSet_multi

[xiab3369]

TestSet仅使用一张参考图像。而正如我们论文中提到的，我们将 4 个引用缝合到一个图像上，与 TTSR 的设置相同。 请尝试以下命令： python test.py --resume './pretrained_weights/masa.pth' --testset TestSet_multi --name masa_TestSet_multi

Actually, TTSR only use ref level=1 image as reference. And we run the code of TTSR with one reference and get the metric as paper persents.

[xiab3369]

TTSR test code,only use ref_level=1

dataset.dataloader.py

class TestSet(Dataset): def init(self, args, ref_level='1', transform=transforms.Compose([ToTensor()])): self.input_list = sorted(glob.glob(os.path.join(args.dataset_dir, 'test/CUFED5', '_0.png'))) self.ref_list = sorted(glob.glob(os.path.join(args.dataset_dir, 'test/CUFED5', '_' + ref_level + '.png'))) self.transform = transform

def __len__(self):
    return len(self.input_list)

def __getitem__(self, idx):
    ### HR
    HR = imread(self.input_list[idx])
    h, w = HR.shape[:2]
    h, w = h//4*4, w//4*4
    HR = HR[:h, :w, :] ### crop to the multiple of 4

trainer:

def evaluate(self, current_epoch=0): self.logger.info('Epoch ' + str(current_epoch) + ' evaluation process...')

    if (self.args.dataset == 'CUFED'):
        self.model.eval()
        with torch.no_grad():
            psnr, ssim, cnt = 0., 0., 0
            for i_batch, sample_batched in enumerate(self.dataloader['test']['1']):
                cnt += 1
                sample_batched = self.prepare(sample_batched)
                lr = sample_batched['LR']
                lr_sr = sample_batched['LR_sr']
                hr = sample_batched['HR']
                ref = sample_batched['Ref']
                ref_sr = sample_batched['Ref_sr']

[SkyeLu]

As shown in Table 4 of TTSR paper, if only the first level reference image is used, the result is 26.99 dB. While as reported in their Table 1, the final result is 27.09 dB. You may refer to this issue of TTSR: https://github.com/researchmm/TTSR/issues/3

[xiab3369]

Thank you for your patient explanation！