search

shaoanlu / faceswap-GAN

A denoising autoencoder + adversarial losses and attention mechanisms for face swapping.
3.37k stars 842 forks source link

Benchmark data set? #85

Open clarle opened 6 years ago

clarle commented 6 years ago

I'm curious if there's a data set being used for benchmarking between the different GAN types (whether quantitatively or qualitatively).

Would love to help contribute but curious to see what you're doing in measuring the differences between them, @shaoanlu.

shaoanlu commented 6 years ago

Most of my experiments were ran on two dataset:

  1. Hinako Sano/Emi Takei images as shown in readme. This dataset is rather clean and well conditioned (contain little duplication). My v1 & v2 models are developed using it. However I don't have plan to release this dataset because of portrait privacy concern.

  2. Shia/Dorney images obtained from issue #50. It is the very in the wild dataset. Fisrt, it contains little training data (~600 for each actor), also, face images are heavily duplicate (600 images cropped from one 21-second-long video) and ill-conditioned as well (e.g., motion blurs, different light condition of source/target faces). The ongoing v2.1 model is developed upon this dataset.

Despite of the above two, some people from deepfakes community are using Trump/Cage dataset, which is included in the original deepfakes source code (?), to show their results. It contains ~300 still images from various sources for each person (instead of from one single video). I haven't tried this dataset yet, but I think this is the most suitable dataset (in terms of variety and file size) for benchmarking among the three.

Nevertheless, I believe there are some more interesting datasets, e.g., 3D avatar to real person, we can collect that are rich of training data and well-conditioned, while the source/target faces should not be too different so that domain transfer/adaption becomes a problem. I feel that establishing a good evaluation/benchmark is important for others to contribute. Do you have any suggestion?

Regarding evaluation, I usually subjectively judge on its visual quality, sometimes convergence speed.

clarle commented 6 years ago

I'm also using the Trump/Cage dataset as well, since it's the one that most people have access to and are familiar with.

@shaoanlu, as a side note, I've seen you mention you using the K80 chip as well. How long does 150 iterations take for you, usually? Using a p2.xlarge instance on AWS, each 150 iterations takes roughly 1300 seconds for me.

shaoanlu commented 6 years ago

I ran my experiments on google colab. V2 model took <1sec/iter w/o perceptual loss. So it sounds like your keras did not use GPU for training.

clarle commented 6 years ago

Found my issue - installing one of the modules (either keras_vggface or face_recognition) causes an older version of Tensorflow to be installed.

If you're using the AWS Deep Learning AMI, this causes Keras to end up using the CPU version of the tensorflow module.

You would need to run:

pip uninstall tensorflow tensorflow-gpu
pip install tensorflow tensorflow-gpu

And then reboot your instance for it to function.

hengshan123 commented 6 years ago

I use my local meachine with tesla m40 to train. the speed is 200 iters / 1200sec. it's so low!

clarle commented 6 years ago

@hengshanji You're probably using CPU like I was before. Try running nvidia-smi to see if you're utilizing your GPU correctly.

For me the problem was Keras was accidentally using the CPU version of the TensorFlow module instead.