search

eladrich / pixel2style2pixel

Official Implementation for "Encoding in Style: a StyleGAN Encoder for Image-to-Image Translation" (CVPR 2021) presenting the pixel2style2pixel (pSp) framework
https://eladrich.github.io/pixel2style2pixel/
MIT License
3.2k stars 568 forks source link

Weird training result #216

Closed dongyun-kim-arch closed 3 years ago

dongyun-kim-arch commented 3 years ago

Hello! Thanks for this amazing model and well documented description.

I am trying to train my own dataset(buildings), but trained results are pretty weird, especially weird pattern covers the whole part. This pattern wasn't shown in my generated images by stylegan model, but it doesn't disappear in 25000epoch.

23800 23200

This is my training dataset and random images generated by stylegan.

tr

Capture2

and this is my training options.

--workers=8 --batch_size=8 --test_batch_size=8 --test_workers=8 --val_interval=2500 --save_interval=5000 --encoder_type=GradualStyleEncoder --start_from_latent_avg --lpips_lambda=0.8 --l2_lambda=1 --id_lambda=0.1

Do you have any insight or thought what's wrong here....?

yuval-alaluf commented 3 years ago

Everything looks good in terms of inputs and models. I've seen these artifacts before, mainly when working on harder domains like yours here. To help with this, I found the MoCo-basrd similarity loss can help (it is similar to the ID similarity loss for faces). If the MoCo loss doesn't improve results, you could look into working with ReStyle which builds on pSp and showed much better results on more challenging domains.

dongyun-kim-arch commented 3 years ago

Thank you for the advice! From your advice, I changed the loss to MoCo loss, and the result is getting better compared to previous one I guess.

I would like to ask you regarding my result. I trained it around half a day and got this result (I am using rtx 3090 as eGPU). It seems that the model captures the overall shape but when I goes into detail of building, it doesn't look great, lost lots of detail.... So, do you think this is due to short training time or there could be some problems such as dataset or domain? If you advice some future direction, that would be so helpful! 27500 27400 27300

Thank you!

yuval-alaluf commented 3 years ago

It could also be that pSp is simply not strong enough to invert and reconstruct these challenging images accurately enough. If you still don't see an improvement, you could check out ReStyle, which showed much better results on non-facial domains. Hopefully, you'll be able to get better reconstructions there.