search

sensity-ai / dot

The Deepfake Offensive Toolkit
BSD 3-Clause "New" or "Revised" License
4.14k stars 410 forks source link

Faceswap on image with Simswap `--use_image` from run_without_camera.md gives incorrect output #12

Closed epicstar7 closed 2 years ago

epicstar7 commented 2 years ago

The faceswap options Using Images from run_without_camera.md gives weird result. Resolution of face wont match the target image or a face is not detected in the image. Is this a known issue?

giorgiop commented 2 years ago

Can you add instructions for reproducing the issue?

epicstar7 commented 2 years ago

No special instruction really here. I Just used the command as described in the file in that exact way on some sample files.

dot --swap_type simswap --target ./data --use_gpu --source "./data" --parsing_model_path ./saved_models/simswap/parsing_model/checkpoint/79999_iter.pth --arcface_model_path ./saved_models/simswap/arcface_model/arcface_checkpoint.tar --checkpoints_dir ./saved_models/simswap/checkpoints --save_folder ./test_local --use_image

epicstar7 commented 2 years ago

I took a closer look at the results, it seems like the whole chin-part of the face is not really swapped at all, that's why the output faces look really strange. Is it to be expected from these models used in this utility that the chin part of the face is not swapped? Does this maybe has something to do with how the model used is exactly trained?

If I look at the faceswap with camera I see the same thing happening, my own chin is basically still in the picture instead of the swapped face's chin. Is this happening to others as well?

ExponentialML commented 2 years ago

@epicstar7 SimSwap works by taking the latent ID of the facial features from the source image, and places it on the target. It doesn't take head features into account during swaps. You would have to train a model for it, as well as modify the SimSwap's training code to get the head features rather than face features. Also, removing the mask parameter or training another face parsing model could help as well if it cuts out too much of the face.

epicstar7 commented 2 years ago

Ok I experimented some more and it turned out that the problem was due to the face in target image being too small. This is quite interesting because when I swap the same images in SimSwap directly there are no problems.

I did notice thought that the output path using _--savefolder is not working correctly, it just saves the files to the main project folder instead of the chosen folder.