search

shepnerd / inpainting_gmcnn

Image Inpainting via Generative Multi-column Convolutional Neural Networks, NeurIPS2018
MIT License
428 stars 97 forks source link

Question: Training the model with a set of images already containing white parts. #63

Open Tenkoni opened 4 years ago

Tenkoni commented 4 years ago

Hello, first of all thank you for your amazing work, as a student it has really helped me understanding GANs and how CNNs can do inpainting. After reading the paper and the code I've come with a question regarding datasets where images are gray (but still formated as RGB). Is the model able to work with datasets where white (255,255,255) is part of the valid information of the image? For what I understand from this part of the paper an image X and a binary region mask M (with value 0 for known pixels and 1 otherwise) and this part of the code image = image * (1-mask) + 255 * mask Such a white space would be considered a hole and inpainted. Is there a way to bypass this? I was wondering if changing the mask from a binary to a RGB one similarly to the ones used in image segmentation would work. Another idea that comes to my mind is modifying the dataset and changing any pure white (255,255,255) for another white, like (254,254,254).

Thank you.

shepnerd commented 4 years ago

Hi Tenkoni, in the inpainting problem, the mask is provided by the user (via some interactive tools) in a binary format independent of the input image. Your mentioned white (255,255,255) in the input in our given images are just for visualization (like image = image * (1-mask) + 255 * mask). The input of the inpainting model is just the image (RGB) and the mask (binary), and the model will try to repair the regions indicated by the mask.

For your later mentioned issue, I am not sure I get your idea.

Tenkoni commented 4 years ago

I see now, thank you! I had the misconception of the mask and the RGB image being operated and the white spaces being interpreted as empty spaces. Would the model be able to work with greyscale (8-bit pixels) images?

shepnerd commented 4 years ago

I see now, thank you! I had the misconception of the mask and the RGB image being operated and the white spaces being interpreted as empty spaces. Would the model be able to work with greyscale (8-bit pixels) images?

Technically, it can. You can just convert the input gray (8 bit) image into a color (24 bit) one first then process it. For opencv, you can read the gray images as color ones if using the default parameters.

Whether it can repair the gray images well depends on the training data. Anyway, you can try the pretrained model first.