How to perform StyleGAN inversion?

[damonzhou]

Hi Yujun,

In the paper you claimed that it must use GAN inversion method to map real images to latent codes, and StyleGAN inversion methods are much better, are there documents introducing how to do the inversion? Any comments are appreciated! Best Regards.

[MDR-EX1000]

Might be a solution https://github.com/Puzer/stylegan-encoder

[ShenYujun]

@MDR-EX1000 Thanks for the answer. That is indeed a solution to StyleGAN model.

@damonzhou Basically, the GAN inversion problem can be solved with "fixing the GAN model, and optimizing the latent code with back propogation to minimize the pixel-wise reconstruction loss or perceptual loss". You can easily achieve this by setting the latent code as the only trainable parameter and running backward function of the deep generator. Hope this answer helps.

[damonzhou]

@MDR-EX1000 @ShenYujun Thanks for the information, I'll try it later and test on Interface-GAN.

[Voyz]

It's not seamless, but I found the following pipeline to work as proof of concept:

1. Download and try-test the stylegan-encoder: https://github.com/Puzer/stylegan-encoder If you're having troubles downloading from Google Drive, copy the StyleGan-ffhq model from InterFaceGAN to stylegan-encoder-master/model/ and change the line 50 in stylegan-encoder-master/encode_images.py to:

   # with dnnlib.util.open_url(URL_FFHQ, cache_dir=config.cache_dir) as f:
   with open(os.path.abspath("model/karras2019stylegan-ffhq-1024x1024.pkl"), 'rb') as f:

2. Create these directories in stylegan-encoder-master: latent_representations aligned_images raw_images generated_images

3. Copy the image you'd like to process to stylegan-encoder-master/raw_images

4. Navigate to stylegan-encoder-master and run the following two commands: python align_images.py raw_images/ aligned_images/ (note you need CMake and dlib) python encode_images.py aligned_images/ generated_images/ latent_representations/

5. The stylegan-encoder should successfully learn the encoding between your image and the latent space. These 'latent codes' should be stored in a file located in stylegan-encoder-master/latent_representations, let's suppose it's called YOUR_LATENT_CODES_FILE.npy Copy that file to InterFaceGAN-master root.

6. Navigate to InterFaceGAN-master root and run the following: python edit.py -m stylegan_ffhq -b boundaries/stylegan_ffhq_age_w_boundary.npy -i YOUR_LATENT_CODES_FILE.npy -o results/my_image_but_aged -s wp

   Note the -s wp, otherwise the dimensions of latent codes won't match and InterFaceGAN will generate 18 faces.

That worked for me. You can play with the boundaries to apply various alterations to the face, as well as the --start and --end distances and --steps.

I suppose that a bespoke mapping working with InterFaceGAN could produce better results. The author of StyleGAN-encoder made a good first step and their code could be an inspiration on how to tackle such task.

It's pretty damn incredible what you guys (authors of all ProGAN, StyleGAN, InterFaceGAN and StyleGAN-encoder) achieved, I praise you and admire you.

[JNash123]

It's not seamless, but I found the following pipeline to work as proof of concept:

1. Download and try-test the stylegan-encoder: https://github.com/Puzer/stylegan-encoder If you're having troubles downloading from Google Drive, copy the StyleGan-ffhq model from InterFaceGAN to stylegan-encoder-master/model/ and change the line 50 in stylegan-encoder-master/encode_images.py to:

    # with dnnlib.util.open_url(URL_FFHQ, cache_dir=config.cache_dir) as f:
    with open(os.path.abspath("model/karras2019stylegan-ffhq-1024x1024.pkl"), 'rb') as f:

2. Create these directories in stylegan-encoder-master: latent_representations aligned_images raw_images generated_images
3. Copy the image you'd like to process to stylegan-encoder-master/raw_images
4. Navigate to stylegan-encoder-master and run the following two commands: python align_images.py raw_images/ aligned_images/ (note you need CMake and dlib) python encode_images.py aligned_images/ generated_images/ latent_representations/
5. The stylegan-encoder should successfully learn the encoding between your image and the latent space. These 'latent codes' should be stored in a file located in stylegan-encoder-master/latent_representations, let's suppose it's called YOUR_LATENT_CODES_FILE.npy Copy that file to InterFaceGAN-master root.
6. Navigate to InterFaceGAN-master root and run the following: python edit.py -m stylegan_ffhq -b boundaries/stylegan_ffhq_age_w_boundary.npy -i YOUR_LATENT_CODES_FILE.npy -o results/my_image_but_aged -s wp Note the -s wp, otherwise the dimensions of latent codes won't match and InterFaceGAN will generate 18 faces.

That worked for me. You can play with the boundaries to apply various alterations to the face, as well as the --start and --end distances and --steps.

I suppose that a bespoke mapping working with InterFaceGAN could produce better results. The author of StyleGAN-encoder made a good first step and their code could be an inspiration on how to tackle such task.

It's pretty damn incredible what you guys (authors of all ProGAN, StyleGAN, InterFaceGAN and StyleGAN-encoder) achieved, I praise you and admire you.

Hello, can i ask u a question? I can get latent code by using encoder network to invert real image, and then put it into pre-train StyleGAN2, I can get reconstruction result. It works well. But I save this latent code as .npy and use : python edit.py -m stylegan_celebahq -b boundaries/stylegan_celebahq_eyeglasses_boundary.npy -i MY_LATENT_CODES_FILE.npy -o results/my_image -s wp

picture is strange, like this: Do i need to use stylegan-encoder after i got the latent codes?

[Jolllly-bot]

It's not seamless, but I found the following pipeline to work as proof of concept:

1. Download and try-test the stylegan-encoder: https://github.com/Puzer/stylegan-encoder If you're having troubles downloading from Google Drive, copy the StyleGan-ffhq model from InterFaceGAN to stylegan-encoder-master/model/ and change the line 50 in stylegan-encoder-master/encode_images.py to:

    # with dnnlib.util.open_url(URL_FFHQ, cache_dir=config.cache_dir) as f:
    with open(os.path.abspath("model/karras2019stylegan-ffhq-1024x1024.pkl"), 'rb') as f:

2. Create these directories in stylegan-encoder-master: latent_representations aligned_images raw_images generated_images
3. Copy the image you'd like to process to stylegan-encoder-master/raw_images
4. Navigate to stylegan-encoder-master and run the following two commands: python align_images.py raw_images/ aligned_images/ (note you need CMake and dlib) python encode_images.py aligned_images/ generated_images/ latent_representations/
5. The stylegan-encoder should successfully learn the encoding between your image and the latent space. These 'latent codes' should be stored in a file located in stylegan-encoder-master/latent_representations, let's suppose it's called YOUR_LATENT_CODES_FILE.npy Copy that file to InterFaceGAN-master root.
6. Navigate to InterFaceGAN-master root and run the following: python edit.py -m stylegan_ffhq -b boundaries/stylegan_ffhq_age_w_boundary.npy -i YOUR_LATENT_CODES_FILE.npy -o results/my_image_but_aged -s wp Note the -s wp, otherwise the dimensions of latent codes won't match and InterFaceGAN will generate 18 faces.

That worked for me. You can play with the boundaries to apply various alterations to the face, as well as the --start and --end distances and --steps. I suppose that a bespoke mapping working with InterFaceGAN could produce better results. The author of StyleGAN-encoder made a good first step and their code could be an inspiration on how to tackle such task. It's pretty damn incredible what you guys (authors of all ProGAN, StyleGAN, InterFaceGAN and StyleGAN-encoder) achieved, I praise you and admire you.

Hello, can i ask u a question? I can get latent code by using encoder network to invert real image, and then put it into pre-train StyleGAN2, I can get reconstruction result. It works well. But I save this latent code as .npy and use : python edit.py -m stylegan_celebahq -b boundaries/stylegan_celebahq_eyeglasses_boundary.npy -i MY_LATENT_CODES_FILE.npy -o results/my_image -s wp

picture is strange, like this: Do i need to use stylegan-encoder after i got the latent codes?

Me too. Please inform me if you find any solution to this problem!