search

zhangzc21 / PetsGAN

22 stars 0 forks source link

Question about refined result(TTSR) #6

Closed junseokoh1 closed 2 years ago

junseokoh1 commented 2 years ago

Hi, I have question about refined result. After train is done, there are results named "hr{num}" and "hr_refined{num}". I guess the difference between these two result is TTSR, which is reference-based image super-resolution model. I can't find any explaination about this from your paper. So if you don't mind could you explain about this? And i have two explicit question. Is it possible to use TTSR 1x scale?(output size is same with input) and is reference image used for TTSR is G.T. without rescaling?

I'd appreciate it if you could leave a comment.

zhangzc21 commented 2 years ago

Hi, @junseokoh1, many thanks for your question. The images with 'hr_refined{num}' are just processed by a patch match module to improve the texture. And you are right, it is really close to the reference-based image super-resolution methods. In the beginning, I refer to the code of ttsr, and rewrite it to efficiently find the nearest neighbor in the pre-trained vgg feature space. Hence the name is a legacy. Although ttsr（or patch match）is a powerful tool, I do not report it in paper. Because I find the module performs unstably when the stride of fold_params is set to 2, and my device does not have enough GPU memory when the stride is set to 1 to produce more delicate results.

For your questions. For the 1x scale, I believe using TTSR, or other reference-based image restoration (including super-resolution) methods is quite possible and helpful (without thinking about the computation). But when the reference image is not rescaled, I believe the difference in scale may cause visual inconsistency (e.g., putting a big balloon in a high-resolution image into a low-resolution image). Thus it may be necessary to ensure that the outputs and reference images have similar scales.

junseokoh1 commented 2 years ago

Thank your for always answering kindly.