style transfer my own artistic portrait generating distorted images

[huanghaifeng1234]

Hi, I followed the training step in Readme and apply the style_transfer.py on content face image but get distorted images, could you help me check the reason.

The script I run

python ./model/stylegan/prepare_data.py --out ./data/portrait/lmdb/ --n_worker 4 --size 1024 ./data/portrait/images/

python -m torch.distributed.launch --nproc_per_node=2 --master_port=8765 finetune_stylegan.py --iter 600 --batch 4 --ckpt ./checkpoint/stylegan2-ffhq-config-f.pt --style portrait --augment ./data/portrait/lmdb/

python destylize.py --model_name finetune-000600.pt --truncation 0.5 --batch 1 --iter 300 portrait

python -m torch.distributed.launch --nproc_per_node=2 --master_port=8765 finetune_dualstylegan.py --iter 1500 --batch 4 --ckpt ./checkpoint/generator-pretrain.pt --style_loss 0.25 --CX_loss 0.25 --perc_loss 0.75 --id_loss 0.75 --L2_reg_loss 0.02 --augment portrait

python refine_exstyle.py --lr_color 0.05 --lr_structure 0.1 --ckpt ./checkpoint/portrait/generator-001400.pt portrait

python style_transfer.py --style portrait --style_id 1 --model_name generator-001400.pt

this is the result image, and the third is the target style portrait, how can I fix the distortion or how to debug it?

And a second question, when I input my own face image but get a big change with respect to the expression and eyes, lips shape, how to skip this reconstruction stage and style-transforming my own face directly?

reconstruction image without align_face

reconstruction image with align_face

[williamyang1991]

Our method is designed for aligned artistic portraits. So your training set should be aligned as the real faces. And you should always use real aligned face as inputs.

For bad reconstruction, you may use the original pSp to predict w+ code or other more advanced stylegan encoder (see https://github.com/williamyang1991/DualStyleGAN/issues/29)

Or you can train a VToonify (https://github.com/williamyang1991/VToonify) based on DualStyleGAN, which should have better reconstruction.

[pavelxx1]

Our method is designed for aligned artistic portraits. So your training set should be aligned as the real faces. And you should always use real aligned face as inputs.

For bad reconstruction, you may use the original pSp to predict w+ code or other more advanced stylegan encoder (see #29)

Or you can train a VToonify (https://github.com/williamyang1991/VToonify) based on DualStyleGAN, which should have better reconstruction.

bro, can u provide small example how and where need to change code, for best reconstruction. I just try few combined flags for generator from #29 and still get poor output to real input photo((

[williamyang1991]

You should not only change flags but also modify the encoder for better reconstruction. These flags are not to make better recontruction, but to match the output of the modified encoder.

[pavelxx1]

You should not only change flags but also modify the encoder for better reconstruction. These flags are not to make better recontruction, but to match the output of the modified encoder.

can u propose any other encoder, which better this one? thx

[williamyang1991]

You could use the original pSp (output w+ latent code) or ReStyle as in https://github.com/williamyang1991/DualStyleGAN/issues/29 Note that I'm not responsible for the debugging on your own encoder.

To make life easy, you may use the improved version of DualStyleGAN -- VToonify (https://github.com/williamyang1991/VToonify) instead, which should have better reconstruction.