Gray region in output

[Ad-Hua]

Thanks for your great work! I have trained your model with my own dataset. However, I found sometimes there will be a gray region in the output, which is the center region of masked area. The attached images is one example.

How can I improve this?

[jingyuanli001]

Hi, I believe this is because the masked region given is too large and the model failed to inpaint the whole area before the iteration number runs out, which then leads to the gray region. This is an architectural limitation of the RFR-Net. The best way to solve this issue is to add one or two extra encoding/decoding layers before and after the RFR module respectively while this would require you to retrain the model. If you don't want to change the model, I would suggest you put the unfinished images and masks into the model again, and let the model inpaint on the incomplete results.

[Ad-Hua]

Thanks for your help. I will follow your advice to see the results.

[Ad-Hua]

Hi! I followed your suggestion but still get the similar result. I add two layers in encoder and two layers in decoder like this: (enc_4): Sequential( (0): Conv2d(512, 512, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False) (1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (2): ReLU(inplace=True) ) (enc_5): Sequential( (0): Conv2d(512, 512, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False) (1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (2): ReLU(inplace=True) (dec_5): Sequential( (0): ConvTranspose2d(1024, 512, kernel_size=(4, 4), stride=(2, 2), padding=(1, 1), bias=False) (1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (2): LeakyReLU(negative_slope=0.2, inplace=True) ) (dec_4): Sequential( (0): ConvTranspose2d(1024, 512, kernel_size=(4, 4), stride=(2, 2), padding=(1, 1), bias=False) (1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (2): LeakyReLU(negative_slope=0.2, inplace=True) )

Will I put more RFRModule help?

[jingyuanli001]

Hi! I followed your suggestion but still get the similar result. I add two layers in encoder and two layers in decoder like this: (enc_4): Sequential( (0): Conv2d(512, 512, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False) (1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (2): ReLU(inplace=True) ) (enc_5): Sequential( (0): Conv2d(512, 512, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False) (1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (2): ReLU(inplace=True) (dec_5): Sequential( (0): ConvTranspose2d(1024, 512, kernel_size=(4, 4), stride=(2, 2), padding=(1, 1), bias=False) (1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (2): LeakyReLU(negative_slope=0.2, inplace=True) ) (dec_4): Sequential( (0): ConvTranspose2d(1024, 512, kernel_size=(4, 4), stride=(2, 2), padding=(1, 1), bias=False) (1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (2): LeakyReLU(negative_slope=0.2, inplace=True) )

Will I put more RFRModule help?

Yes, putting more RFR modules definitely helps. The encoding layer in the code above should be composed of Partial convolution layers which updates the mask m, instead of the normal convolution layers which do not update the mask.

[Ad-Hua]

Thanks for your help! I tried more Pconv and RFR modules and the problem was solved.

[qwerdbeta]

@Ad-Hua can you please share your code? I am trying to add these layers but getting some problems. Thanks!

[Ad-Hua]

Hi! I change the number of RFR modules from 6 to 10 and solve the problem. As for adding Pconv, I added like this:

self.Pconv21 = PartialConv2d(64, 64, 7, 1, 3, multi_channel = True, bias = False) self.Pconv22 = PartialConv2d(64, 64, 7, 1, 3, multi_channel = True, bias = False)

---

self.Pconv23 = PartialConv2d(64, 64, 3, 1, 1, multi_channel = True, bias = False) self.Pconv24 = PartialConv2d(64, 64, 3, 1, 1, multi_channel = True, bias = False)

---

This can help to release that problem but cannot solve totally . So I suggest you to add RFR modules if you dont require a high speed inference.

[qwerdbeta]

thanks a lot!

[qwerdbeta]

@Ad-Hua it works. When I made the change you suggested, it seems to make the filled in area more fine grain. It's still early in the iterations so I'm still waiting to see how it comes out as it converges more. Are our images larger than 256x256? they look large ?

[Ad-Hua]

Hi! @qwerdbeta I assume you are asking whether the gray region become larger if the input mask is larger. In my opinion, Yes. So you may change some parameters to get good results.

[qwerdbeta]

@Ad-Hua No, I meant your target_size as defined in run.py.

Also, I noticed that you can get rid of the grey areas by increasing the values below from:

        self.Pconv1 = PartialConv2d(3, 64, 7, 2, 3, multi_channel = True, bias = False)
        self.bn1 = nn.BatchNorm2d(64)
        self.Pconv2 = PartialConv2d(64, 64, 7, 1, 3, multi_channel = True, bias = False)
        self.bn20 = nn.BatchNorm2d(64)
        self.Pconv21 = PartialConv2d(64, 64, 7, 1, 3, multi_channel = True, bias = False)
        self.Pconv22 = PartialConv2d(64, 64, 7, 1, 3, multi_channel = True, bias = False)

to:

    self.Pconv1 = PartialConv2d(3, 64, 7, 2, 3, multi_channel = True, bias = False)
    self.bn1 = nn.BatchNorm2d(64)
    self.Pconv2 = PartialConv2d(64, 64, 9, 1, 4, multi_channel = True, bias = False)
    self.bn20 = nn.BatchNorm2d(64)
    self.Pconv21 = PartialConv2d(64, 64, 9, 1, 4, multi_channel = True, bias = False)
    self.Pconv22 = PartialConv2d(64, 64, 9, 1, 4, multi_channel = True, bias = False)

and even more to:

    self.Pconv1 = PartialConv2d(3, 64, 7, 2, 3, multi_channel = True, bias = False)
    self.bn1 = nn.BatchNorm2d(64)
    self.Pconv2 = PartialConv2d(64, 64, 11, 1, 5, multi_channel = True, bias = False)
    self.bn20 = nn.BatchNorm2d(64)
    self.Pconv21 = PartialConv2d(64, 64, 11, 1, 5, multi_channel = True, bias = False)
    self.Pconv22 = PartialConv2d(64, 64, 11, 1, 5, multi_channel = True, bias = False)

However, I have not made this change because I fear it has other effects. I have not figured out what those are though. I am worried about losing very high level line patterns that move left to right.

@jingyuanli001 do you know what other effects these changes have?

[qwerdbeta]

Actually the larger kernal size (9 or 11) might get larger features which i want, but I'm not sure what the padding increases do or if they are necesary

[qwerdbeta]

What if we used larger kernal sizes and padding for the 2 new Pconv layers? The code could be like this to get the Pconv23 and Pconv24 layers to learn larger patterns from combinations of the smaller kernal ones before it?

    self.Pconv1 = PartialConv2d(3, 64, 7, 2, 3, multi_channel = True, bias = False)
    self.bn1 = nn.BatchNorm2d(64)
    self.Pconv2 = PartialConv2d(64, 64, 7, 1, 3, multi_channel = True, bias = False)
    self.bn20 = nn.BatchNorm2d(64)
    self.Pconv21 = PartialConv2d(64, 64, 7, 1, 3, multi_channel = True, bias = False)
    self.Pconv22 = PartialConv2d(64, 64, 7, 1, 3, multi_channel = True, bias = False)
    self.Pconv23 = PartialConv2d(64, 64, 9, 1, 5, multi_channel = True, bias = False)
    self.Pconv24 = PartialConv2d(64, 64, 9, 1, 5, multi_channel = True, bias = False)

...or would it be better to put the larger kernal sizes first to first get large patterns then smaller ones like this:

    self.Pconv1 = PartialConv2d(3, 64, 7, 2, 3, multi_channel = True, bias = False)
    self.bn1 = nn.BatchNorm2d(64)
    self.Pconv2 = PartialConv2d(64, 64, 9, 1, 5, multi_channel = True, bias = False)
    self.bn20 = nn.BatchNorm2d(64)
    self.Pconv21 = PartialConv2d(64, 64, 9, 1, 5, multi_channel = True, bias = False)
    self.Pconv22 = PartialConv2d(64, 64, 9, 1, 5, multi_channel = True, bias = False)
    self.Pconv23 = PartialConv2d(64, 64, 3, 1, 1, multi_channel = True, bias = False)
    self.Pconv24 = PartialConv2d(64, 64, 3 1, 1, multi_channel = True, bias = False)

[qwerdbeta]

@Ad-Hua my other question is this: when you added Pconv23 and Pconv24, did you also add them to the forward function? Pconv21 and Pconv22 are added in the lines:

    for i in range(10):
        x2, m2 = self.Pconv21(x2, m2)
        x2, m2 = self.Pconv22(x2, m2)

[SSTTfit]

@qwerdbeta Have you figured out the problem?About your two questions.

[mychina75]

I'v got this result. similiar with yours?