Figure: Image editing results using ReSeFa on BigGAN (first two columns) and StyleGAN2 (last three columns).
Region-Based Semantic Factorization in GANs
Jiapeng Zhu, Yujun Shen, Yinghao Xu, Deli Zhao, Qifeng Chen
International Conference on Machine Learning (ICML) 2022
[Paper] [Project page] [Demo] [Colab]
In the repository, we propose a simple algorithm to interpret the region-based semantics learned by GANs. In particular, we re-examine the task of local editing with pre-trained GAN models, and formulate region-based semantic discovery as a dual optimization problem. Through an appropriately defined generalized Rayleigh quotient, we are able to tackle such a problem super efficiently with a closed-form solution. Extensive experiments on BigGAN and StyleGAN2 demonstrate the effectiveness and robustness of our proposed method.
Figure: Semantics found by ReSeFa with respect to the region of interest (i.e., within the green boxes).
This repository is based on Hammer, where you can find detailed instructions on environmental setup. Please refer here for model downloading and conversion, and use the script synthesis.py
to check whether the model is ready for use as
MODEL_PATH='stylegan2-ffhq-config-f-1024x1024.pth'
python synthesis.py ${MODEL_PATH} --stylegan2
We have provided some local semantic directions under the directory directions/
, which are discovered from the officially released StyleGAN2/StyleGAN3 models. Users can use these directions for image local editing. For example,
Manipulation on StyleGAN2 FFHQ
MODEL_PATH='stylegan2-ffhq-config-f-1024x1024.pth'
DIRECTION='directions/ffhq/stylegan2/eyesize.npy'
python manipulate.py ${MODEL_PATH} ${DIRECTION} --stylegan2
Manipulation on StyleGAN3-R AFHQ
MODEL_PATH='stylegan3-r-afhqv2-512x512.pth'
DIRECTION='directions/afhq/stylegan3/eyes-r.npy'
python manipulate.py ${MODEL_PATH} ${DIRECTION} \
--stylegan3 --cfg R --img_size 512 --data_name afhq
To explore more directions, please follow the following steps, which use the FFHQ model officially released in StyleGAN2 as an example.
Step-1: Compute Jacobian with one/several random syntheses.
MODEL_PATH='stylegan2-ffhq-config-f-1024x1024.pth'
python compute_jacobian.py ${MODEL_PATH} --stylegan2
Step-2: Discover semantic directions from the Jacobian.
JACOBIAN_PATH='work_dirs/jacobians/ffhq/jacobians_w.npy'
python compute_directions.py ${JACOBIAN_PATH}
Step-3: Verify the directions via image manipulation
MODEL_PATH='stylegan2-ffhq-config-f-1024x1024.pth'
DIRECTION_PATH='work_dirs/directions/ffhq/eyes/${DIRECTION_NAME}'
python manipulate.py ${MODEL_PATH} ${DIRECTION} --stylegan2
@inproceedings{zhu2022resefa,
title = {Region-Based Semantic Factorization in {GAN}s},
author = {Zhu, Jiapeng and Shen, Yujun and Xu, Yinghao and Zhao, Deli and Chen, Qifeng},
booktitle = {International Conference on Machine Learning (ICML)},
year = {2022}
}