search

Zhaoyi-Yan / Shift-Net_pytorch

Pytorch implementation of Shift-Net: Image Inpainting via Deep Feature Rearrangement (ECCV, 2018)
http://openaccess.thecvf.com/content_ECCV_2018/papers/Zhaoyi_Yan_Shift-Net_Image_Inpainting_ECCV_2018_paper.pdf
MIT License
363 stars 83 forks source link

Issues with mask size #106

Open walbermr opened 4 years ago

walbermr commented 4 years ago

Hello, I was studying your work and during some tests I found that the current approach has issues with the size of masks: if there is no mask at all, the shift will try to find a mask in the latent space, and as it does not have a case to no mask in the latent space, the code it will crash. This case can be expanded to a case where the mask is small enough to be compressed, generating no mask in some reduction, during the compression phase, crashing again.

What modifications can be done to remove that issue? Currently I am inserting a mask with 8x8 pixels in a irrelevant part of the image, but it is not optimal. As I still have not thought in a better solution, I am asking you a better approach. I am open to develop that and make a pull request in your repo.

Zhaoyi-Yan commented 4 years ago

Wow, it is truly a bug. However, I am afraid such small masks deverse no shift at all. In fact, the information flow of convolution from the outside to the inside are propagating little by little. When the mask is very big, it is hard for the network to direclty predict the missing content. In this case, shift operation makes the inpainting adpots the information not only local context but also global one. However, when the mask is very small, the shift is not necessary at all. So you can direcly skip shift operaition when mask is missing in the reduction.

walbermr commented 4 years ago

Indeed. So, I will code the fix and re-run some tests to make a pull request later. Thanks for the fast answer!