How to finetune on cats and generate toonification on animals.

[Zeeshan75]

Hi @yuval-alaluf,

Thanks for your quick response.

For toonification on cats. Below is the approach I thought, correct me if I am wrong.

1. Train the hyperstyle gan model to generate the latent space on for the given input cat image.
2. using that latent space use toonification/finetuning with new cartoon cat dataset to generate cartoon cat images.

Below are some of the issues I am facing: I thought of using the pre-trained AFHQ_wild hypernet model to generate the latent space, But the trained model is more biased towards fox and tiger than cats, so I thought of finetuning the model using the AFHQ_wild model on my new cat's dataset. Below are the training parameters I am considering, Pls let me know if I am doing wrong, I am getting below error.

Error:

Loading hypernet weights from resnet34! Loading decoder weights from pretrained path: /home/ubuntu/Desktop/Avatar-gen/hyper_z/hyperstyle/pretrained_models/hyperstyle_afhq_wild.pt Traceback (most recent call last): File "/home/ubuntu/Desktop/Avatar-gen/hyper_z/hyperstyle/scripts/train.py", line 32, in main() File "/home/ubuntu/Desktop/Avatar-gen/hyper_z/hyperstyle/scripts/train.py", line 19, in main coach = Coach(opts) File "/home/ubuntu/Desktop/Avatar-gen/hyper_z/hyperstyle/./training/coach_hyperstyle.py", line 35, in init self.net = HyperStyle(self.opts).to(self.device) File "/home/ubuntu/Desktop/Avatar-gen/hyper_z/hyperstyle/./models/hyperstyle.py", line 25, in init self.load_weights() File "/home/ubuntu/Desktop/Avatar-gen/hyper_z/hyperstyle/./models/hyperstyle.py", line 58, in load_weights self.decoder.load_state_dict(ckpt['g_ema'], strict=True) KeyError: 'g_ema'

====================================================================== I am using the below training parameters.

python scripts/train.py \ --dataset_type=my_hypernet \ --encoder_type=SharedWeightsHyperNetResNet \ --exp_dir=experiments/hyperstyle \ --workers=4 \ --batch_size=4 \ --test_batch_size=2 \ --test_workers=2 \ --val_interval=5000 \ --save_interval=10000 \ --lpips_lambda=0.8 \ --l2_lambda=1 \ --id_lambda=0 \ --n_iters_per_batch=5 \ --max_val_batches=150 \ --output_size=256 \ --stylegan_weights=pretrained_models/hyperstyle_afhq_wild.pt \ --load_w_encoder \ --w_encoder_checkpoint_path pretrained_models/afhq_wild_w_encoder.pt \ --layers_to_tune=0,2,3,5,6,8,9,11,12,14,15,17,18,20,21,23,24

For toonification: I have the cartoon cat dataset around 400 images, How to finetune after I got the latents for the given input cat image?

[yuval-alaluf]

I could not reproduce your error with the official AFHQ wild checkpoint in this repo. I was able to successfully load it using the most recent code. Did you make any changes to the file/code?

[Zeeshan75]

Actually, I did some of the changes, while loading from state_dict, at some places I replaced it with "strict = False", I will try to again clone and will try it. Any approach/ steps you can share to apply toonification on cats?

[yuval-alaluf]

For the toonification of cats, you will first need a generator that can generate toons of cats. After that, I guess I would try using the domain adaptation approach provided in the repo. You could try to generate the cat toons generator using StyleGAN-NADA.

[Zeeshan75]

Thanks, @yuval-alaluf for your suggestion.