New pre-trained GAN 128x128 image module from TF Hub

[fat-tire]

Anyone watch yesterday's Tensorflow summit video introducing pre-trained "modules" that can be imported into TF and retrained? They mentioned this module "Progressive GAN trained on CelebA for 128x128 images," aka google/progan-128/1 (the 1 is the version number) pre-trained on faces which they demo "morphing" from one face to another.

I wonder if this module might be a good generic "starting model" for faceswap that might help save time or in cases where there are fewer training images? I guess it may be best to have a pre-baked module similar to the faces you're training with, but this might be a good common starting point...

Thoughts? Related paper.

[bryanlyon]

Pretrained models are generally more work than starting a specific model from scratch. While there are some gains in "generic" I.E. Many faces to one (like done with Cage), it is less useful for one to one faceswapping due to the amount of extra training they require. Remember, also that any model that is used will be stuck to that model so we can't experiment with other model shapes with a pretrained. Finally, most models are hundreds of megabytes in size, and including such large binaries would make the project very difficult to download and share. For these reasons, I doubt that we will use pretrained modules as a major part of this project soon, but if we find a good use for them, I see no reason why we would exclude them.

[bryanlyon]

That said, you are welcome to download the pretrained model and experiment with it. If you find good results, let us know, we can include a link to it in the instructions.

[fat-tire]

Fair enough. Just wanted to put this on everyone's radar. Didn't mean to imply it would replace the current models, just that it might be added to the /plugins as an option to start training with the common face module from Google and build off that.

I don't think the model size is a big deal, as the module is not included in the repository-- the "mod" concept is comparable in some way to a library in regular programming-- it's external to the main program. The video describes that it's downloaded at runtime/buildtime (I'm assuming cached locally) and only a simple link to the module is included in the code.

If I get a chance I'll try it out, but wanted to at least put it out there that this is an option now.

[bryanlyon]

Of course, and if you issue a PR which provides an option to download that, we can use it. My only concern was that you were meaning to include the actual model in the package, which would be a large binary file in the git repo. Even just including it as a default "add on" through a script would give the project excessively long downloads when most people wouldn't be using it.

We'd also need to make sure that the model architecture they're using matched our GAN128 (or another module if necessary). Since a model trained on a different shape would be useless if we don't match it's shape.

[fat-tire]

No no download :) this is something new as of yesterday that google announced, though it may require Tensorflow 1.7 to support it- not sure. It's brand new. I'll submit a PR if I have a chance to poke at it, but of course anyone else is welcome to try it out and see how it works, how well and fast it trains etc compared to starting from scratch.

(And hopefully it wouldn't d/l anything unless the plugin was pulled in/active.)

[deepfakesclub]

A "simpler" alternative that fits the current pipeline would be for somebody to train the current algorithms (original, gan, whatever) for 1 week on the CelebA dataset and share the resulting models with everyone. It's doable... just some shell scripts to rotate through 10K training images at a time. CelebA is ~200K total.

I set out to do this but realized that the CelebA dataset is largely a waste of time for basic faceswapping after looking through it. (Maybe useful for other training such as superresolution.)

It is derived from celebrity photos, which contain very few angles other than standard portrait poses. If you do not have enough images, what you are lacking is usually a variety of poses. Throwing in more of the same front face poses doesn't help.

At the other end of the spectrum, some people re-use old models for new faces. I hate this, as the results are always contaminated with the old facial features. Some people pre-train 10 faces into a starting model, which probably helps a bit, but I don't think 10 faces is enough.

Videos are a better source material to extract faces with all angles and expressions in volume. I wanted to compile a universal starting model using a better curated training data set. But first, this would require a good way to sort and maximize face diversity in image sets.

[fat-tire]

That's actually a cool idea-- to create a universal faceswap starter module based on a generalized face training dataset (trained from contributions from the community?) There's no reason these modules have to be hosted at google, right? Could use something like:

embed = hub.Module("https://github.com/deepfakes/releases/swapmodel-GAN128/1")

or whatever. As someone with an older GPU that doesn't even support CUDA, this would save hours of training I'd assume.

[Madhivarman]

@fat-tire That's really a cool idea to use pre-trained model for this task. I ran this model from scratch on the cloud but it gets stopped after 18,100 epochs. I tested face swap in that trained model and the result is not that convincing. Training a GAN model on pre-trained weights is a time saver and if you able to share sample code, it would be a great help for us or please upload/share your pre-trained model.

[kvrooman]

Closing old issue