About model's result on cropped pictures from CelebA dataset

jingyuanli001 / RFR-Inpainting

The source code for CVPR 2020 accepted paper "Recurrent Feature Reasoning for Image Inpainting"

MIT License

355 stars 76 forks source link

About model's result on cropped pictures from CelebA dataset #26

Closed ZhaoWeicheng98 closed 3 years ago

ZhaoWeicheng98 commented 3 years ago

First of all, thank you for your work!

I tried the CelebA pretrained weight on some pictures that were cropped from pictures in the CelebA dataset. These cropped pictures are mostly of faces. However, I found that though the model worked well on pictures originally from the CelebA dataset, it did not give out a good result when cropped pictures were provided. Masks are randomly added to my tests.

For example, here are 2 pictures from the original CelebA dataset. 000006 000007

here are the masked of these 2 pictures masked_img_4 masked_img_5

The model's results on these 2 pictures are indeed good.

Here are my cropped version of those 2 pictures. I chose the face parts and cropped them out. face000006 face000007

here are the masked of these 2 cropped pictures masked_img_14 masked_img_15

However,the results this time are not very good, even they were masked to a similar extent.

Considering the result, the model might be overfitting? Looking forward to your reply on this test.

jingyuanli001 commented 3 years ago

Hi, thanks for your question first! For this question, I thought it is due to the biases in the training images in the dataset, i.e. they put the face in the center of of an image. Then the model might only learn to inpaint images under this setting. Besides, it seems that the cropped images are also resized to to 256, under which setting the model might fail due to the change of context resolution. Can you try to inpaint on the cropped images without resizing them? I thought there should have some way to generate good results even if the images are cropped.

ZhaoWeicheng98 commented 3 years ago

Thank you for your quick reply. I tried setting the target size to 128*128 on the cropped images following your instruction. However, it seems better than before, still not very well.

Masked: masked_img_14 masked_img_15

Result:

If the set target size is 128*128 for original CelebA images, the results are worse than before.

Masked: masked_img_4 masked_img_5

Result:

Maybe a re-train or finetune is needed for a cross-resolution generation? Thanks!

jingyuanli001 commented 3 years ago

Thank you for your quick reply. I tried setting the target size to 128*128 on the cropped images following your instruction. However, it seems better than before, still not very well.

Masked:

Result:

If the set target size is 128*128 for original CelebA images, the results are worse than before.

Masked:

Result:

Maybe a re-train or finetune is needed for a cross-resolution generation? Thanks!

Yeah, given the results from your case, it then seems to be necessary to at least finetune the model for a while.