search

deepfakes / faceswap-playground

User dedicated repo for the faceswap project
306 stars 194 forks source link

Utilizing more of the raws in GAN conversion? #226

Closed IEWbgfnYDwHRoRRSKtkdyMDUzgdwuBYgDKtDJWd closed 5 years ago

IEWbgfnYDwHRoRRSKtkdyMDUzgdwuBYgDKtDJWd commented 5 years ago

Not really an issue, so figured id post here and not main repo.

But lately I have been using GAN, and have been actually impressed despite the generally negative experience people tend to have. Yes, it does not produce favorable results for anything realistic compared to original models, but what I have found is that it does extremely well with making extreme facial angles and expressions come out more lifelike. I will watch the raws, and see some really great ones mixed in, ones where the face is up and has a strange face that is nothing like my dataset faces, almost like its a bad jpeg of the person. Then I will go to convert, and it only adds a piece of jaw or eyebrow here and there.

I know this is the nature of GAN and masking/occlusions. But sometimes the raws will look so good, i wish it would just swap all the learning. I have even taken raws and polished them on another machine, and fed those units into my training dataset believe it or not, just a few here and there, nothing too lossy. it actually helps a lot.

Anyone else found any useful ways to utilize the unique nature of the GAN results?

bryanlyon commented 5 years ago

What you refer to here is called "augmenting" your data with "synthetic data". It's common in AI research and is a great way to build more resilient models. GAN as it exists in our codebase is inefficient and poor for general faceswap, but I'm glad you've found it helpful.

If you're looking at more synthetic data that may help you you may want to look at face models. Some of these can work with as few as a single picture and give you a face model you can rotate to unique angles. In addition, with the face model there is no background to distract the model. https://github.com/YadiraF/PRNet https://github.com/unibas-gravis/parametric-face-image-generator

These experiments can help you understand what an AI is doing and I highly recommend spending some time seeing what you can do. But it's important to know that synthetic data has flaws. It tends to amplify errors and cause difficulty with learning.

This is definitely the correct place for questions like this. You can also join us in our Discord server at https://discord.gg/rPVZ4W if you want to have more of a discussion.