Implementation of the PixelUnshuffle layer

[SimonKoppers]

Hi all,

I just stumbled accross your paper and tried to use your architecture for a RGB -> NIR prediction. I'm a little confused by the PixelUnshufflingLayer implented in your network.

Isn't this equivalent to Reshape->Permute->Reshape of the input tensor?

As an example: PixelUnShuffle input: batch * 3 * 256 * 256 x1 = self.pixelUnshuffle0(x_in) # out: batch * 12 * 128 * 128 x2 = self.pixelUnshuffle1(x_in) # out: batch * 48 * 64 * 64 x3 = self.pixelUnshuffle2(x_in) # out: batch * 192 * 32 * 32

should be equivalent to: PixelUnShuffle input: batch * 3 * 256 * 256 x1 = x_in.reshape(-1, 3, 128, 2, 128, 2).permute(0, 1, 3, 5, 2, 4).reshape(-1, 12, 128, 128) # out: batch * 12 * 128 * 128 x2 = x_in.reshape(-1, 3, 64, 4, 64, 4).permute(0, 1, 3, 5, 2, 4).reshape(-1, 48, 64, 64) # out: batch * 48 * 64 * 64 x3 = x_in.reshape(-1, 3, 32, 8, 32, 8).permute(0, 1, 3, 5, 2, 4).reshape(-1, 192, 32, 32) # out: batch * 192 * 32 * 32

I evaluated the runtime using the following script (using a RTX 2080 Ti):

from time import time x_in = torch.rand(1, 3, 256, 256).cuda() t0 = time() numTrials = 10000 for idx in range(0, numTrials): x1 = self.pixelUnshuffle0(x_in) # out: batch * 12 * 128 * 128 x2 = self.pixelUnshuffle1(x_in) # out: batch * 48 * 64 * 64 x3 = self.pixelUnshuffle2(x_in) # out: batch * 192 * 32 * 32 t1 = time() print('Evaluation took ' + str((t1-t0)/numTrials) + ' seconds per shuffle block using the PixelUnshuffle layer.')

Evaluation took 0.005510963249206543 seconds per shuffle block using the PixelUnshuffle layer. using a batchsize of 128: Evaluation took 0.01887419834136963 seconds per shuffle block using the PixelUnshuffle layer.

and:

from time import time x_in = torch.rand(1, 3, 256, 256).cuda() t0 = time() numTrials = 10000 for idx in range(0, numTrials): x1 = x_in.reshape(-1, 3, 128, 2, 128, 2).permute(0, 1, 3, 5, 2, 4).reshape(-1, 12, 128, 128) # out: batch * 12 * 128 * 128 x2 = x_in.reshape(-1, 3, 64, 4, 64, 4).permute(0, 1, 3, 5, 2, 4).reshape(-1, 48, 64, 64) # out: batch * 48 * 64 * 64 x3 = x_in.reshape(-1, 3, 32, 8, 32, 8).permute(0, 1, 3, 5, 2, 4).reshape(-1, 192, 32, 32) # out: batch * 192 * 32 * 32 t1 = time() print('Evaluation took ' + str((t1-t0)/numTrials) + ' seconds per shuffle block using reshape, permute and reshape again.')

Evaluation took 0.0001397745132446289 seconds per shuffle block using reshape, permute and reshape again. using a batchsize of 128: Evaluation took 0.0012832509517669679 seconds per shuffle block using reshape, permute and reshape again.

Is there a reason to use a PixelUnshuffle layer for this operation, as reshape, permute and reshape again seems to realize the same at a much fast rate?

[SimonKoppers]

to make sure that both ways result in the same output, I used the following script: x = torch.rand(128, 3, 256, 256).cuda() print(torch.abs(self.pixelUnshuffle0(x) - x.reshape(-1, 3, 32, 8, 32, 8).permute(0, 1, 3, 5, 2, 4).reshape(-1, 192, 32, 32)).sum()) print(torch.abs(self.pixelUnshuffle1(x) - x.reshape(-1, 3, 64, 4, 64, 4).permute(0, 1, 3, 5, 2, 4).reshape(-1, 48, 64, 64)).sum()) print(torch.abs(self.pixelUnshuffle0(x) - x.reshape(-1, 3, 128, 2, 128, 2).permute(0, 1, 3, 5, 2, 4).reshape(-1, 12, 128, 128)).sum())

This shows that the results are the same.

[zhaoyuzhi]

Hello, thanks for your sharing. Actually, the layer is same to reshape -> permute -> reshape operation.

Could you please try the new ones:

class PixelShuffle(nn.Module):
    def __init__(self, upscale_factor):
        super(PixelShuffle, self).__init__()
        self.upscale_factor = upscale_factor

    def forward(self, x):
        if len(x.size()) != 4:
            raise ValueError("input tensor shape {} is not supported.".format(x.size()))
        N, C, H, W = x.size()
        c = C // (self.upscale_factor ** 2)
        h, w = H * self.upscale_factor, W * self.upscale_factor
        # (N, C, H, W) => (N, c, r, r, H, W)
        x = x.reshape(-1, c, self.upscale_factor,
                        self.upscale_factor, H, W)
        x = x.permute(0, 1, 4, 2, 5, 3)
        x = x.reshape(-1, c, h, w)
        return x

class PixelUnShuffle(nn.Module):
    def __init__(self, downscale_factor):
        super(PixelUnShuffle, self).__init__()
        self.downscale_factor = downscale_factor

    def forward(self, x):
        if len(x.size()) != 4:
            raise ValueError("input tensor shape {} is not supported.".format(x.size()))
        N, C, H, W = x.size()
        c = int(C * (self.downscale_factor ** 2))
        h, w = H // self.downscale_factor, W // self.downscale_factor
        x = x.reshape(-1, C, h, self.downscale_factor, w, self.downscale_factor)
        x = x.permute(0, 1, 3, 5, 2, 4)
        x = x.reshape(-1, c, h, w)
        return x

[SimonKoppers]

I just revaluated the Unshuffle: the old layer: 0.01917232780456543 seconds per unshuffle block (batch size 128) the new layer: 0.0013062042951583863 seconds per unshuffle block (batch size 128) without a layer: 0.0012821023225784301 seconds per unshuffle block (batch size 128)

the new layer seems to be much better, as it basically does the same as the line I wrote.

concerning the PixelShuffle: your implementation: 0011729759216308594 seconds per shuffle block (batch size 128) pytorch internal implemenation: 0.0011687211751937865 seconds per shuffle block (batch size 128)

[zhenghuizhao]

不好意思，提问写错了位置。 我只是想测试下，与GAN方法相比，IR->RGB的效果。 谢谢你的回复，这好像不是我需要的。

------------------ 原始邮件 ------------------ 发件人: "zhaoyuzhi/Hierarchical-Regression-Network-for-Spectral-Reconstruction-from-RGB-Images" <notifications@github.com>; 发送时间: 2020年9月8日(星期二) 晚上7:56 收件人: "zhaoyuzhi/Hierarchical-Regression-Network-for-Spectral-Reconstruction-from-RGB-Images"<Hierarchical-Regression-Network-for-Spectral-Reconstruction-from-RGB-Images@noreply.github.com>; 抄送: "zzh"<904150289@qq.com>;"Comment"<comment@noreply.github.com>; 主题: Re: [zhaoyuzhi/Hierarchical-Regression-Network-for-Spectral-Reconstruction-from-RGB-Images] Implementation of the PixelUnshuffle layer (#1)

I think you need this: https://github.com/zhaoyuzhi/Hierarchical-Regression-Network-for-Spectral-Reconstruction-from-RGB-Images/blob/master/official%20scoring%20code/clean_example.ipynb

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe.