search

eladrich / pixel2style2pixel

Official Implementation for "Encoding in Style: a StyleGAN Encoder for Image-to-Image Translation" (CVPR 2021) presenting the pixel2style2pixel (pSp) framework
https://eladrich.github.io/pixel2style2pixel/
MIT License
3.2k stars 568 forks source link

Find the latent code of the synthetic image which is identical to the real image in some region? #204

Closed tommy-qichang closed 3 years ago

tommy-qichang commented 3 years ago

Thank you for your great work. We are doing some experiments based on your code. But we need to find the latent code that the synthetic images are identical to the original one in some regions (we need the latent code of course). So far the PSP model provides a very similar images overall, but we could easily find the differences for the ROI.

I think there are several options(1. disable noises for each level in generator-stylegan. 2.put more weight on the ROI region, etc), but not sure if I missed any, or it's an impossible mission. Could you please shed some light for me? Thanks.

yuval-alaluf commented 3 years ago

Unfortunately finding the latent code that will return the exact same image is of course not possible. If all you care about is the reconstruction quality then you should take a look at optimization techniques such as those from StyleGAN2 or a recent work called PTI. Those optimization techniques will result in much better reconstruction but will take several minutes per image. In addition, some optimization techniques (mainly the latent vector optimization techniques) result in latents that are not editable so the latent itself is not very useful.

Regarding the options you mentioned:

  1. Disabling noise: this is already done during inference. We kept noise during training because we found it did not affect the training process and it kind of acted as an interesting regularization to the training process.
  2. You can try adding more weight to the region of interest (e.g., the inner face region), but even with the higher weight, you will not lead to perfect reconstructions. It would also hurt the reconstruction of regions outside that region (e.g., the background), which will also hurt the results.

So to summarize: if you want the best reconstruction possible go with optimization techniques, which do have their own drawbacks. Just note that you won't be able to get a near-perfect reconstruction.

tommy-qichang commented 3 years ago

I know it's impossible to return the exact same image, I just wondering if we could find the same shape for the specific area. Let me give an example. I want to find a synthetic image that has a similar cardiac shape to the real one(for instance the left ventricle myocardium). The rest of the image may not necessarily be the same. Overall I think the synthetic image looks really good, but you could see the read arrow part is not as the same shape as the real one. Real: image Synthetic image

yuval-alaluf commented 3 years ago

The results seem very good to be honest. You could try optimization techniques to try to achieve slightly better reconstruction, but overall I am not sure how much more improvement you will be able to achieve