How to apply the generated 1D kernels to the detail layer?

[zhs1]

Hi, Thanks for your great work, I have a question when implementing it with pytorch: How to apply the generated 1D kernels to the detail layer? Could you explain it in detail? Thank you!

[sooyekim]

Hi @zhs1 ,

Thanks for the interest in our work! I apologize for the late reply.

The resulting filter block (one for vertical and one for horizontal) from the Detail Restoration subnet would be a 3-dimensional tensor of size HxWx(kxscalexscale). If we assume a case for scale=2, there would be four blocks of HxWxk stacked in the channel dimension.

• Local filtering operation: For each of these blocks, the 1x1xk vector at each (h, w) grid location is considered as a local filter to be applied on the 1xk (or kx1) neighborhood at each (h, w) grid location. By applying the filter, I mean the inner product between the 1xk (or kx1) neighborhood at each (h, w) position of the detail layer (decomposed from the input frame), and each 1x1xk 1D kernel.
• Separable filtering operation: After applying all the vertical 1D filters, the horizontal 1D filters are applied in the same way. (The order doesn't matter as long as you use the same order for training and testing)
• Upsampling operation: The above operations are repeated for all four blocks of size HxWxk. Then, the result would be a tensor of size HxWx(scalexscale), which can be reordered using pixel shuffle to (Hxscale)x(Wxscale) for the upsampling part.

I tried to be as descriptive as possible but if any part of the explanation is unclear, please do not hesitate to ask. Also, you could refer to the uploaded Tensorflow version of the code now that it's updated :)

Best, Soo Ye