Reversed Mask Loss

[sergiocasaspastor]

Hi, congratulations on such amazing work.

I have a couple of doubts about the code in the generator_loss function. The paper says that "reversed mask loss compares the squared difference for corresponding pixels specific for regions created by the mask on the image and the reconstructed pixels of the masked-image."

However, when computing the reversed_mask_loss, the reverse mask (inverse of the mask) is used, comparing only the pixels out of the regions created by the mask. This makes this loss always 0. Therefore, I think maybe we should use directly the mask instead of the inverse of the mask, to compare only the regions inside the mask.

Mask

Inverse mask

If I use this inverse mask, the loss is comparing regions outside of the mask, which always gives 0 for me. If I use the mask, then this reverse mask loss gives a value above 0.

The second doubt is about the weight of each loss in the generator. The paper says that the optimal is for λ=0.4 on reversed mask loss. However, in the code you use 0.4perceptual_loss + 0.6reversed_mask_loss, which is the opposite.

I hope you understand my doubts to reproduce the results.

Thank you very much.

[mdelhussieny]

I think he means that the masked pixel value will be zero so he used revers mask

[Jireh-Jam]

The reverse mask loss compares generated regions only. It does not compare the regions outside the mask. The pixel regions you want to keep is 1 and the mask regions is 0. When you reverse the mask on the image, the mask pixel regions become 1 and the rest of the image pixel regions become 0. However, I have made changes to the generator loss. If you are trying to reproduce the code your own way, you have to make sure the reversed mask when applied on the image shows only the valid regions. Be sure to view the image before proceeding to compile and run the code. Otherwise follow the instructions. I will upload some pertained weights by the first week of January 2022.

[Jireh-Jam]

def generator_loss(self, y_true,y_pred):
    ones = K.ones_like(y_pred[:,:,:,1])
    mask = K.stack([ones]*self.channels, axis=-1)
    input_img = Lambda(lambda x : x[:,:,:,0:3])(y_true)
    output_img = Lambda(lambda x : x[:,:,:,0:3])(y_pred)
    reversed_mask = Lambda(self.reverse_mask,output_shape=(self.img_shape_mask))(mask)
    vgg = VGG19(include_top=False, weights='imagenet', input_shape=self.img_shape)
    loss_model = Model(inputs=vgg.input, outputs=vgg.get_layer('block3_conv3').output)
    loss_model.trainable = False
    p_loss = K.mean(K.square(loss_model(output_img) - loss_model(input_img)))
    masking = Multiply()([reversed_mask,input_img])
    predicting = Multiply()([reversed_mask, output_img])
    reversed_mask_loss = (K.mean(K.square(loss_model(predicting) - loss_model(masking))))
    new_loss = 0.6*(p_loss) + 0.4*reversed_mask_loss
    return new_loss         

[Jireh-Jam]

Pre-trained model is ready. You can use this for inpainting.

[mdelhussieny]

Thanks