SSIM from the paper.

[tomashlehmann]

Hello, Thank you for your work and great dataset! I have a question about the SSIM from your paper. I've calculated this metric with your methods (from utils/metric.py) between images from the test_origin data set and SSIM oscillated around 0.90. In your paper the value is almost twice smaller. When you take a look at the date it also doesn't look like the SSIM ~ 50 is alright. How actually have you calculated it (which images were compared)?

Best,

[XinYu-Andy]

Hi, Thanks for your interest in our work. Could you explain the question more clearly? What do you mean SSIM ~50? The "test_origin" data contains the full 4K resolution images （~4032x3024) which are not used for evaluation (we just release them for researchers' convenience). We evaluate the data in "test" folder which contains standard 4K resolution images (i.e., 3840x2160).

Hello, Thank you for your work and great dataset! I have a question about the SSIM from your paper. I've calculated this metric with your methods (from utils/metric.py) between images from the test_origin data set and SSIM oscillated around 0.90. In your paper the value is almost twice smaller. When you take a look at the date it also doesn't look like the SSIM ~ 50 is alright. How actually have you calculated it (which images were compared)?

Best,

[tomashlehmann]

I meant SSIM ~= 0.50, of course, how it is written in Table 2. of your paper. Isn't the resolution the only difference between "test" and "test_origin" data? If so, I assume that the SSIM should be pretty similar (still far from 0.90).

[XinYu-Andy]

I guess maybe you forget to set the " LOAD_PATH" in the configuration file. Which checkpoint did you used for evaluation? Have you checked the output images, what does it look like?

I meant SSIM ~= 0.50, of course, how it is written in Table 2. of your paper. Isn't the resolution the only difference between "test" and "test_origin" data? If so, I assume that the SSIM should be pretty similar (still far from 0.90).

[tomashlehmann]

I actually don't strictly use your code. I was just checking SSIM and PSNR between input images (the pair of images from test_origin/test dataset). I've loaded the data and used the create_metrics class to check these values. Do you somehow modify data before checking metrics?

[XinYu-Andy]

If what you mean is the SSIM between input image and ground-truth, I test it through replacing the output image of the model by the input image (as shown in the following code in the ./model/model.py).

I actually don't strictly use your code. I was just checking SSIM and PSNR between input images (the pair of images from test_origin/test dataset). I've loaded the data and used the create_metrics class to check these values. Do you somehow modify data before checking metrics?

[tomashlehmann]

I do it in the same way to calculate input metrics. I compare the pair of images nnnn_gt.jpg and nnnn_moire.jpg and then calculate SSIM with your metrics. But achieved results are far different. Do you use some pre-processing to regress the quality of moire image? I wrote a class which returns two tensor representations of the ground truth and moire image. And later I call create_metric functions to calculate metrics from these tensors. Average SSIM is around 0.90 which seems wrong but idk why. Did you calculate metrics from the raw images or did you change something in them?

[XinYu-Andy]

Have you transformed the tensor from int [0,255] into float tensor [0,1]?

I do it in the same way to calculate input metrics. I compare the pair of images nnnn_gt.jpg and nnnn_moire.jpg and then calculate SSIM with your metrics. But achieved results are far different. Do you use some pre-processing to regress the quality of moire image? I wrote a class which returns two tensor representations of the ground truth and moire image. And later I call create_metric functions to calculate metrics from these tensors. Average SSIM is around 0.90 which seems wrong but idk why. Did you calculate metrics from the raw images or did you change something in them?

[tomashlehmann]

My getitem looks like below. self.x_images are the lists with the file path.

    def __getitem__(self, idx):
        sharp_image = self.sharp_images[idx]
        moire_image = self.moire_images[idx]

        hr_img = cv2.imread(str(sharp_image), 1)
        lr_img = cv2.imread(str(moire_image), 1)

        lr_img = cv2.resize(lr_img, (self.img_size, self.img_size))
        hr_img = cv2.resize(hr_img, (self.img_size, self.img_size))

        return (
            torch.from_numpy(np.transpose(lr_img, (2, 0, 1))).type("torch.FloatTensor"),
            torch.from_numpy(np.transpose(hr_img, (2, 0, 1))).type("torch.FloatTensor")
        )

[XinYu-Andy]

Can you print the value and the size of "torch.from_numpy(np.transpose(lr_img, (2, 0, 1))).type("torch.FloatTensor")".

My getitem looks like below. self.x_images are the lists with the file path.

    def __getitem__(self, idx):
        sharp_image = self.sharp_images[idx]
        moire_image = self.moire_images[idx]

        hr_img = cv2.imread(str(sharp_image), 1)
        lr_img = cv2.imread(str(moire_image), 1)

        lr_img = cv2.resize(lr_img, (self.img_size, self.img_size))
        hr_img = cv2.resize(hr_img, (self.img_size, self.img_size))

        return (
            torch.from_numpy(np.transpose(lr_img, (2, 0, 1))).type("torch.FloatTensor"),
            torch.from_numpy(np.transpose(hr_img, (2, 0, 1))).type("torch.FloatTensor")
        )

[tomashlehmann]

The size is (3, 768, 768). If I change resolution the metrics are pretty same (approximately).

[XinYu-Andy]

I suggest you transform the dynamic range [0,255] into dynamic range [0,1]. We test the "SSIM" code in this dynamic range and I am not sure if the codebase works for int range. You can refer to the SSIM implementation (used in our codebase for UHDM dataset) from https://github.com/jorge-pessoa/pytorch-msssim.

The size is (3, 768, 768). If I change resolution the metrics are pretty same (approximately).

[tomashlehmann]

I did it too right now and for resolution 3840x2160 results are: PSRN = 17.07 SSIM = 0.57

Much better but still kinda far from the paper's SSIM value (0.51).

[XinYu-Andy]

I have tested it today for several times and the metric results are the same with those in the paper. I suggest you directly use our codebase for avoiding inconsistent results for comparisons. In my experience, the metrics' calculation sometimes get not the exactly same results due to many reasons (float or int, yuv or rgb, different python built-in packages), especially for SSIM.
This is why I follow previous works' implementations on other benchmarks for fairness, as you can see this from my code's annotations in metric.py.

I did it too right now and for resolution 3840x2160 results are: PSRN = 17.07 SSIM = 0.57

Much better but still kinda far from the paper's SSIM value (0.51).

I did it too right now and for resolution 3840x2160 results are: PSRN = 17.07 SSIM = 0.57

Much better but still kinda far from the paper's SSIM value (0.51).

[tomashlehmann]

Ok. Thank you for your help ;) I will check it out.

[undcloud]

goog job