Unofficial Pytorch implementation (inference only) of the SimSwap: An Efficient Framework For High Fidelity Face Swapping

Updates

• improved performance (up to 40% in some scenarios, it depends on frame resolution and number of swaps per frame).
• fixed a problem with overlapped areas from close faces (https://github.com/mike9251/simswap-inference-pytorch/issues/21)
• added support for using GFPGAN model as an additional post-processing step to improve final image quality
• added a toy gui app. Might be useful to understand how different pipeline settings affect output

Attention

This project is for technical and academic use only. Please do not apply it to illegal and unethical scenarios.

In the event of violation of the legal and ethical requirements of the user's country or region, this code repository is exempt from liability.

Preparation

Installation

# clone project
git clone https://github.com/mike9251/simswap-inference-pytorch
cd simswap-inference-pytorch

# [OPTIONAL] create conda environment
conda create -n myenv python=3.9
conda activate myenv

# install pytorch and torchvision according to instructions
# https://pytorch.org/get-started/

# install requirements
pip install -r requirements.txt

Important

Face detection will be performed on CPU. To run it on GPU you need to install onnx gpu runtime:

pip install onnxruntime-gpu==1.11.1

and modify one line of code in ...Anaconda3\envs\myenv\Lib\site-packages\insightface\model_zoo\model_zoo.py

Here, instead of passing None as the second argument to the onnx inference session

class ModelRouter:
    def __init__(self, onnx_file):
        self.onnx_file = onnx_file

    def get_model(self):
        session = onnxruntime.InferenceSession(self.onnx_file, None)
        input_cfg = session.get_inputs()[0]

pass a list of providers

class ModelRouter:
    def __init__(self, onnx_file):
        self.onnx_file = onnx_file

    def get_model(self):
        session = onnxruntime.InferenceSession(self.onnx_file, providers=['CUDAExecutionProvider', 'CPUExecutionProvider'])
        input_cfg = session.get_inputs()[0]

Otherwise simply use CPU onnx runtime with only a minor performance drop.

Weights

Weights for all models get downloaded automatically.

You can also download weights manually and put inside weights folder:

• weights/face_detector_scrfd_10g_bnkps.onnx
• weights/arcface_net.jit
• weights/79999_iter.pth
• weights/simswap_224_latest_net_G.pth - official 224x224 model
• weights/simswap_512_390000_net_G.pth - unofficial 512x512 model (I took it here).
• weights/GFPGANv1.4_ema.pth
• weights/blend_module.jit

Inference

Web App

streamlit run app_web.py

Command line App

This repository supports inference in several modes, which can be easily configured with config files in the configs folder.

• replace all faces on a target image / folder with images

  python app.py --config-name=run_image.yaml

• replace all faces on a video

  python app.py --config-name=run_video.yaml

• replace a specific face on a target image / folder with images

  python app.py --config-name=run_image_specific.yaml

• replace a specific face on a video

  python app.py --config-name=run_video_specific.yaml

Config files contain two main parts:

• data

  • id_image - source image, identity of this person will be transferred.
  • att_image - target image, attributes of the person on this image will be mixed with the person's identity from the source image. Here you can also specify a folder with multiple images - identity translation will be applied to all images in the folder.
  • specific_id_image - a specific person on the att_image you would like to replace, leaving others untouched (if there's any other person).
  • att_video - the same as att_image
  • clean_work_dir - whether remove temp folder with images or not (for video configs only).

• pipeline

  • face_detector_weights - path to the weights file OR an empty string ("") for automatic weights downloading.
  • face_id_weights - path to the weights file OR an empty string ("") for automatic weights downloading.
  • parsing_model_weights - path to the weights file OR an empty string ("") for automatic weights downloading.
  • simswap_weights - path to the weights file OR an empty string ("") for automatic weights downloading.
  • gfpgan_weights - path to the weights file OR an empty string ("") for automatic weights downloading.
  • device - whether you want to run the application using GPU or CPU.
  • crop_size - size of images SimSwap models works with.
  • checkpoint_type - the official model works with 224x224 crops and has different pre/post processings (imagenet like). Latest official repository allows you to train your own models, but the architecture and pre/post processings are slightly different (1. removed Tanh from the last layer; 2. normalization to [0...1] range). If you run the official 224x224 model then set this parameter to "official_224", otherwise "none".
  • face_alignment_type - affects reference face key points coordinates. Possible values are "ffhq" and "none". Try both of them to see which one works better for your data.
  • smooth_mask_kernel_size - a non-zero value. It's used for the post-processing mask size attenuation. You might want to play with this parameter.
  • smooth_mask_iter - a non-zero value. The number of times a face mask is smoothed.
  • smooth_mask_threshold - controls the face mask saturation. Valid values are in range [0.0...1.0]. Tune this parameter if there are artifacts around swapped faces.
  • face_detector_threshold - values in range [0.0...1.0]. Higher value reduces probability of FP detections but increases the probability of FN.
  • specific_latent_match_threshold - values in range [0.0...inf]. Usually takes small values around 0.05.
  • enhance_output - whether to apply GFPGAN model or not as a post-processing step.

Overriding parameters with CMD

Every parameter in a config file can be overridden by specifying it directly with CMD. For example:

python app.py --config-name=run_image.yaml data.specific_id_image="path/to/the/image" pipeline.erosion_kernel_size=20

Video

License

For academic and non-commercial use only.The whole project is under the CC-BY-NC 4.0 license. See LICENSE for additional details.

Acknowledgements

• SimSwap
• Insightface
• Face-parsing.PyTorch
• BiSeNet
• GFPGAN