Questions about the model

[mrgreen3325]

Thanks for your great work. I have several questions about the model. For the training, may I know that the DIV2k HR is used for I_hr and my camera photos are used for I_lr (which I_hr and I_lr are completely different photos), is that right? Also, is that the ADL learn the kernel from I_lr (my camera photos), and apply that kernel to downsample the I_hr(DIV2K HR) into a downsample version (DIV2K HR)? In the paper, is that the Oracle experiment use the paired RealSR HR and LR to train, which the HR and LR are the real one, not downsample one? Thanks a lot.

[JaehaKim97]

Hi @mrgreen3325,

1) Yes, your understanding is correct. In our main manuscript, we set DIV2K HR set as I_HR, then set arbitrary LR images set as I_LR.

2) Yes, our adaptive downsampling model learns the corresponding downsampling kernel of I_LR. (More precisely, we aim to find the downsampling kernel which simulates I_HR to the I_LR.)

3) Yes. In the case of the Oracle setting in the RealSR dataset, we report the PSNR of the SR model which is trained with the official HR and LR pairs from the RealSR dataset.

Feel free to ask me if you have further questions!

Jaeha

[mrgreen3325]

Hi @mrgreen3325,

1. Yes, your understanding is correct. In our main manuscript, we set DIV2K HR set as I_HR, then set arbitrary LR images set as I_LR.
2. Yes, our adaptive downsampling model learns the corresponding downsampling kernel of I_LR. (More precisely, we aim to find the downsampling kernel which simulates I_HR to the I_LR.)
3. Yes. In the case of the Oracle setting in the RealSR dataset, we report the PSNR of the SR model which is trained with the official HR and LR pairs from the RealSR dataset.

Feel free to ask me if you have further questions!

Jaeha

Thanks Jaeha. I have some Samsung s21 photos. If I input these photos as I_LR and DIV2k as I_HR, will the ADL generate a downsampled Samsung style DIV2K LR photos?

[JaehaKim97]

I think so if you have enough LR samples.

Make sure that LR samples are not corrupted with other degradations such as noise or compression artifacts, as our method is not focused to handle them.

Jaeha

[mrgreen3325]

I think so if you have enough LR samples.

Make sure that LR samples are not corrupted with other degradations such as noise or compression artifacts, as our method is not focused to handle them.

Jaeha

Thanks Jaeha. I have 2000 pieces s21 2k photos, is it enough for that? For the target LR photos, is it better to use some small size image, will 2k too big for that? Also, how long did your downsample training last for ? I notice that the downsampling training only has 80 epochs, is it enough?

[JaehaKim97]

Yes, that amount of photos seems to be enough. You can also consider using a small subset of images through random sampling for faster training.

For resolution, using 2k as the target LR domain may be too large as our source domain DIV2K is also almost 2k images, but it depends on the situation so I recommend you to try various configurations.

Yes, training with 80 epochs was enough for our situation. Checking the stability of training by observing the shape of the retrieved downsampling kernel and then adjusting the training epoch will also be a good option.

Jaeha

[mrgreen3325]

Yes, that amount of photos seems to be enough. You can also consider using a small subset of images through random sampling for faster training.

For resolution, using 2k as the target LR domain may be too large as our source domain DIV2K is also almost 2k images, but it depends on the situation so I recommend you to try various configurations.

Yes, training with 80 epochs was enough for our situation. Checking the stability of training by observing the shape of the retrieved downsampling kernel and then adjusting the training epoch will also be a good option.

Jaeha

Thanks Jaeha. May I know that does the code can randomly sampling subset of images? Also, does the source and target both be crop into small patches for training as I got 22425 data for each epoch while I only have 2000 2k LR, and DIV2K HR? I got the estimated kernel for epoch 14, and it is a bit different to yours.

Is that normal?

[JaehaKim97]

No, the code currently does not support the automated construction of a subset of images. When we test our model on the DPED dataset, we manually set 120 randomly sampled images for the target LR.

Actually, __len__ of the dataset is determined as

where (w, h) is (width, height) of each image, and p is determined patch_size. So your number seems reasonable to me.

ADL is first applied in 10 epochs and then iteratively updated at every 10 epochs, so the kernel shape of 14 epochs may be unstable. BTW, we have observed that the proposed model was able to achieve better performance than the bicubic SR model even with that kind of strange kernel shape.

[mrgreen3325]

No, the code currently does not support the automated construction of a subset of images. When we test our model on the DPED dataset, we manually set 120 randomly sampled images for the target LR.

Actually, __len__ of the dataset is determined as

where (w, h) is (width, height) of each image, and p is determined patch_size. So your number seems reasonable to me.

ADL is first applied in 10 epochs and then iteratively updated at every 10 epochs, so the kernel shape of 14 epochs may be unstable. BTW, we have observed that the proposed model was able to achieve better performance than the bicubic SR model even with that kind of strange kernel shape.

Thanks for reply. Is that the dped are 2k images? Does the ADM only learn from the LR images during the training (the HR images only for down_results)? Also, I completed the training in 100 epoches. However, the generated LR (down_results) has some color drift and artifact. The generated LR: The bicubic LR:

The estimated kernel in epoch 100:

May I have some clue to fix it?

[JaehaKim97]

As far as I know, DPED is consist of 2048x1536 resolution images.

No, ADM requires both HR images and LR images for training. Note that the ADM learns the downsampling process from the given HR images set to the given LR images set.

Check below two conditions that have to be satisfied for ADM training.

(1) Is your personal LR images set mainly consisting of face data as your example? Scene variation of LR images can also be a critical issue since it can be an easy discriminative feature rather than a downsampling kernel of LR images.

(2) Color drift may be from the data bias. Check if your LR image set is biased to the specific color domains. If then, you can consider below two kinds of solutions.

(2-1) Newly construct LR images set without specific color bias. (2-2) Use HR images that have a similar color domain as your LR images.

I hope this answer is helpful for your work.

Jaeha

[mrgreen3325]

As far as I know, DPED is consist of 2048x1536 resolution images.

No, ADM requires both HR images and LR images for training. Note that the ADM learns the downsampling process from the given HR images set to the given LR images set.

Check below two conditions that have to be satisfied for ADM training.

(1) Is your personal LR images set mainly consisting of face data as your example? Scene variation of LR images can also be a critical issue since it can be an easy discriminative feature rather than a downsampling kernel of LR images.

(2) Color drift may be from the data bias. Check if your LR image set is biased to the specific color domains. If then, you can consider below two kinds of solutions.

(2-1) Newly construct LR images set without specific color bias. (2-2) Use HR images that have a similar color domain as your LR images.

I hope this answer is helpful for your work.

Jaeha

Thanks Jaeha. No. I don't have any face data in LR, only HR contain face data. Do you mean that LR images have lack of some color? Those LR are randomly taken by the phone camera, I did not have bias to take. Did you experience that color shift before?

[JaehaKim97]

(We discussed more issues in an e-mail.)

I close this issue but feel free to ask me further questions.

Jaeha