Hello, Chief Scientist. would you like publicize the image inpainting model?

[GuardSkill]

I am a postgraduate student in China, I find a new partial-conv based (mask based ) convolution method for some special inpainting application. I have been re-implementing your inpainting model for a long time. But it doesn't get a good result as yours. Could u help me to publicize the official model code or send to me? I am hurrying to publish my first paper by modifying your method. Thanks very much

[liuguilin1225]

@GuardSkill You can take a look at the third-party implementation of our paper at https://github.com/MathiasGruber/PConv-Keras or some other implementations on github. The differences are:

1. all the L1 losses need to be size-averaged.
2. some minor difference between the partial convolution layer implements (mathematically there are the same; just some implementation trick difference). You can check both codes. If you want me to run some experiments for partial conv based inpainting, please send me your images and masks.

[GuardSkill]

Thanks to your answer. The paper didn't mention the size-averaged in loss term, are ur experimental hyperparameters(loss term weights) conducted by size-averaged loss term that even include total variation?
And I see https://github.com/MathiasGruber/PConv-Keras use a strange 'normalization' variable to replace , Is it conflict ur original paper? I think ur paper do this in each sliding windows rather than the whole feature map. The one layers convolution code as follows:

input[0] : image/feature map input[1]: mask map self.kernel_mask: ones_like kernel

    def call(self, inputs, mask=None):
        '''
        We will be using the Keras conv2d method, and essentially we have
        to do here is multiply the mask with the input X, before we apply the
        convolutions. For the mask itself, we apply convolutions with all weights
        set to 1.
        Subsequently, we set all mask values >0 to 1, and otherwise 0
        ''' 

        # Both image and mask must be supplied
        if type(inputs) is not list or len(inputs) != 2:
            raise Exception('PartialConvolution2D must be called on a list of two tensors [img, mask]. Instead got: ' + str(inputs))

        # Create normalization. Slight change here compared to paper, using mean mask value instead of sum
        normalization = K.mean(inputs[1], axis=[1,2], keepdims=True)
        normalization = K.repeat_elements(normalization, inputs[1].shape[1], axis=1)
        normalization = K.repeat_elements(normalization, inputs[1].shape[2], axis=2)

        # Apply convolutions to image
        img_output = K.conv2d(
            (inputs[0]*inputs[1]) / normalization, self.kernel, 
            strides=self.strides,
            padding=self.padding,
            data_format=self.data_format,
            dilation_rate=self.dilation_rate
        )

        # Apply convolutions to mask
        mask_output = K.conv2d(
            inputs[1], self.kernel_mask, 
            strides=self.strides,
            padding=self.padding,            
            data_format=self.data_format,
            dilation_rate=self.dilation_rate
        )

        # Where something happened, set 1, otherwise 0        
        mask_output = K.cast(K.greater(mask_output, 0), 'float32')

        # Apply bias only to the image (if chosen to do so)
        if self.use_bias:
            img_output = K.bias_add(
                img_output,
                self.bias,
                data_format=self.data_format)

        # Apply activations on the image
        if self.activation is not None:
            img_output = self.activation(img_output)

        return [img_output, mask_output]

[liuguilin1225]

@GuardSkill Yes; we were using size-averaged loss for total variation as well. Somehow, total variation loss doesn't contribute a lot. The sum(M) should be computed over each sliding window. https://arxiv.org/pdf/1811.11718.pdf has a better explanation of how partial conv layer works in section 3.

[ezavarygin]

@liuguilin1225 Sorry for interrupting your discussion. What is "size" in size-averaging in your loss calculations? Do you average the losses over the mini-batch axis only, i.e. divide them by the mini-batch size B? Or, do you divide them by the total number of elements in the sum: B*W*H*C (and B*C*C when using with Gram matrices)?

[liuguilin1225]

@ezavarygin The size-averaging loss is averaged over the total number of elements, which is BWH*C as you mentioned. You can check the new version of the arxiv paper: https://arxiv.org/pdf/1804.07723.pdf

[ezavarygin]

Great, thanks! I was working with the first version of the paper.

[DongHwanJang]

@GuardSkill Wanna ask you whether re-implementation was successful or not. Has Keras repo shown decent output, compared with Nvidia API? Or were there any other repos helpful? Great Thx.