Question about default image color values in hole area

[hi-zhengcheng]

Hi Knazeri, the code of getting images_masked is as follows:

• in EdgeModel:

  images_masked = (images * (1 - masks).float()) + masks

• in InpaintingModel:

  images_masked = (images * (1 - masks).float()) + masks

the hole areas in the image are filled with 1. I also read some other inpainting code and find that some code filled holes with 0. Is the inpainting result affected by filling hole areas with 1 or 0?

[knazeri]

@hi-zhengcheng Honestly, our motivation for filling the hole with 1 was visualization and the ease of use on inference. I don't believe one could expect much difference using 1 or 0. Having said that, when the missing region is filled with 0, then for those regions completely covering the convolution filter the output will be zero and hence the gradient is zero; while in principle convolving 1s results in some gradient for training.

[hi-zhengcheng]

@knazeri , thanks for your reply. I have another question about discriminator's input. For example, in InpaintingModel model:

outputs = self(images, edges, masks)
...
dis_input_fake = outputs.detach()
dis_fake, _ = self.discriminator(dis_input_fake)
...
gen_input_fake = outputs
gen_fake, _ = self.discriminator(gen_input_fake)

The outputs represents the fake images created by the generator, and it includes all area(mask area and other area). Then the discriminator guides the generator create the same pixel values like in the ground truth images. I don't know if it's better to let discriminator only guide generator create good pixel values in mask area. Or may be create good pixel values in whole area will help to create good pixel values in mask area.

So my question is, to the discriminator, will it be better to make the input as:

discriminator(outputs * mask + images * (1 - mask))

Could you share some opinions?

[knazeri]

@hi-zhengcheng This is what we discussed a lot when first designing the model. In principle, it only makes sense to have the discriminator only discriminate the pixel values in the mask area. However in practice, this makes the discriminator too powerful! One explanation might be that there might be some subtle discontinuity around the edge when merging the output with the ground truth surrounding! The discriminator then learns to pick the edges around the mask border and the learning gets stalled!

We decided to go with the actual output of the generator instead. Also in case you haven't noticed, our generator is also penalized by the perceptual loss on the entire image which makes its output for non-mask areas very similar to the ground truth, and the input to the discriminator is very similar to what you expect!

[hi-zhengcheng]

@knazeri Thanks for your reply. Great work!