search

Lotayou / Face-Renovation

Official repository of the paper "HiFaceGAN: Face Renovation via Collaborative Suppression and Replenishment".
https://arxiv.org/abs/2005.05005
GNU General Public License v3.0
284 stars 49 forks source link

Image artifacts #12

Closed Darrenxc closed 4 years ago

Darrenxc commented 4 years ago

Hi, I followed your instructions to train the network, However, there are some artifacts in generated images These artifacts normally happen on where relatively more white pixel locate at, such as eyes, tip of nose. there are some pics for details
epoch023_iter227400_input_label epoch023_iter227400_real_image epoch023_iter227400_synthesized_image

epoch023_iter228800_synthesized_image

Can I ask the reason why these artifacts happened, and how can I solve these. Thank you!

Lotayou commented 4 years ago

I could use more information, how did you process the training data, with what type of degradation? Is the testing data prepared the same way as the training data?

From what I gather, the issue is most likely inherited from the original SPADE network. In fact, we observed even more severe artifacts when using the vanilla SPADE with face parsing maps as the input (Never mind the 16x mosaic on the left column, I just haven't got time to change the visualization script yet): image image

and here is the renovation result using our network, where the input is the left 16x mosaic image, you can still notice the unusual white dot on the woman's nose tip, but for most cases the result is acceptable: image image

As to the exact cause of such artifact, I think it could be similar as the ones reported in StyleGAN2, and maybe applying the corresponding fix could help (for instance, remove the bias normalization). Let me know if you have any progress:)

Darrenxc commented 4 years ago

following the stylegan2 instructions about removing instance norm can eliminate this artifact, I also use mobilev2,v3 to compress this model and face parsing as an additional input, but I notice that you have mentioned a bug in your code that the predicted images are not shown as expected when we set the model to eval model. Do you find a way to solve this problem? I try to fine-tune the entire model for a large of epochs with a very low learning rate, However, it still not good as I expected.

Lotayou commented 4 years ago 
but I notice that you have mentioned a bug in your code that the predicted images are not shown as expected when we set the model to eval model.

This is a Pytorch bug where the spectral normalization layer in SPADE is not working under eval mode in dataparallel. A possible fix is suggested at here