search

eladrich / pixel2style2pixel

Official Implementation for "Encoding in Style: a StyleGAN Encoder for Image-to-Image Translation" (CVPR 2021) presenting the pixel2style2pixel (pSp) framework
https://eladrich.github.io/pixel2style2pixel/
MIT License
3.19k stars 570 forks source link

face segmentation questions #278

Closed skunkwerk closed 2 years ago

skunkwerk commented 2 years ago

Thank you for a wonderful repository & support.

Is there a recommended method for generating the segmentation maps of a face? Ie. is there a specific mapping of colors to face segments that the model supports?

Also, if I generate segmentation maps from each frame of a video of a face, and ran inference on each segmentation map, would the generated faces be consistent in terms of identity over time?

yuval-alaluf commented 2 years ago

Note that our model is really designed for segmentation. There are many works that focus on the task of face segmentation. For example, this is the project we used to get our paired data: https://github.com/switchablenorms/CelebAMask-HQ

There are also some interesting works that try to generate faces with corresponding face masks. For example, SemanticGAN: https://nv-tlabs.github.io/semanticGAN/ I believe this is more in-line to what you are looking for.

Regarding generating an image from a segmentation video, it is not guaranteed that the identity will be consistent between all the frames. Most importantly, you need to make sure that when performing style mixing, you always use the same latent code to perform the mixing on the fine layers. My best advice is to try running inference on a short video and see what you get :)