Region-Based Semantic Factorization in GANs

Figure: Image editing results using ReSeFa on BigGAN (first two columns) and StyleGAN2 (last three columns).

Region-Based Semantic Factorization in GANs
Jiapeng Zhu, Yujun Shen, Yinghao Xu, Deli Zhao, Qifeng Chen
International Conference on Machine Learning (ICML) 2022

[Paper] [Project page] [Demo] [Colab]

In the repository, we propose a simple algorithm to interpret the region-based semantics learned by GANs. In particular, we re-examine the task of local editing with pre-trained GAN models, and formulate region-based semantic discovery as a dual optimization problem. Through an appropriately defined generalized Rayleigh quotient, we are able to tackle such a problem super efficiently with a closed-form solution. Extensive experiments on BigGAN and StyleGAN2 demonstrate the effectiveness and robustness of our proposed method.

Local Face Editing Results

Figure: Semantics found by ReSeFa with respect to the region of interest (i.e., within the green boxes).

Usage

Pre-trained Models

This repository is based on Hammer, where you can find detailed instructions on environmental setup. Please refer here for model downloading and conversion, and use the script synthesis.py to check whether the model is ready for use as

MODEL_PATH='stylegan2-ffhq-config-f-1024x1024.pth'
python synthesis.py ${MODEL_PATH} --stylegan2

Editing with Provided Directions

We have provided some local semantic directions under the directory directions/, which are discovered from the officially released StyleGAN2/StyleGAN3 models. Users can use these directions for image local editing. For example,

• Manipulation on StyleGAN2 FFHQ

  MODEL_PATH='stylegan2-ffhq-config-f-1024x1024.pth'
  DIRECTION='directions/ffhq/stylegan2/eyesize.npy'
  python manipulate.py ${MODEL_PATH} ${DIRECTION} --stylegan2

• Manipulation on StyleGAN3-R AFHQ

  MODEL_PATH='stylegan3-r-afhqv2-512x512.pth'
  DIRECTION='directions/afhq/stylegan3/eyes-r.npy'
  python manipulate.py ${MODEL_PATH} ${DIRECTION} \
    --stylegan3 --cfg R --img_size 512 --data_name afhq

Finding Your Own Directions

To explore more directions, please follow the following steps, which use the FFHQ model officially released in StyleGAN2 as an example.

• Step-1: Compute Jacobian with one/several random syntheses.

  MODEL_PATH='stylegan2-ffhq-config-f-1024x1024.pth'
  python compute_jacobian.py ${MODEL_PATH} --stylegan2

• Step-2: Discover semantic directions from the Jacobian.

  JACOBIAN_PATH='work_dirs/jacobians/ffhq/jacobians_w.npy'
  python compute_directions.py ${JACOBIAN_PATH}

• Step-3: Verify the directions via image manipulation

  MODEL_PATH='stylegan2-ffhq-config-f-1024x1024.pth'
  DIRECTION_PATH='work_dirs/directions/ffhq/eyes/${DIRECTION_NAME}'
  python manipulate.py ${MODEL_PATH} ${DIRECTION} --stylegan2

BibTeX

 @inproceedings{zhu2022resefa,
  title     = {Region-Based Semantic Factorization in {GAN}s},
  author    = {Zhu, Jiapeng and Shen, Yujun and Xu, Yinghao and Zhao, Deli and Chen, Qifeng},
  booktitle = {International Conference on Machine Learning (ICML)},
  year      = {2022}
}