A few requests for image and video face swapping inference

[dasasf]

Hello. First off, let me congratulate you on your great repo, it is amazing what it can already do. However, I ran into a couple of issues, some small, and hopefully easy to fix, and some larger ones.

1) Right now, swapping a single image creates the output file "result_whole_swapmulti.jpg", which requires renaming it manually and gets annoying quickly.

Would it be possible to have the output file be named as "title_dst_Picture - title_src_picture.jpg" ? In the demo_file case, the output file would be named "iron_man-specific1.jpg" for example.

2) Additionally, an option to have a whole folder of pictures either as src, dst or both would be nice.

3) Face detection has trouble with profile shots, quick camera movement and other angles different from portrait shots. Would it be possible to update insightface to a newer version ? Assuming it would help of course.

For multiface video swaps:

4) When trying to swap multiple faces, the same person person is often not recognized, especially when the distance to the camera changes or in profile shots. You can mitigate it by running the face swap again using new src/dst pictures with the previous output video but it is time consuming and doesn't always work.

Now, if I understand it correctly, Insightface can output what it believes are different faces, as seen here:

https://camo.githubusercontent.com/94afea3fd9149d1859c601e5f396b2d57ed803d8cc1401fcbdc76e669e162943/68747470733a2f2f696e7369676874666163652e61692f6173736574732f696d672f6769746875622f666163657265636f676e6974696f6e66726f6d766964656f2e504e47

Would it be possible to have an option to run a preliminary face recognition over a video/folder and get a src picture for every different face in the video ? Ideally one could then assign one dst face to all src faces the program thinks are different but are actually the same person.

Thank you for your time and, once again, thank you for the amazing work you have already done.

[mata3o]

hey @dasasf could you tell me what commands you use for swapping single images? i cannot for the life of me get image swap to work no matter what while video works just fine

[dasasf]

What does the error message say when you try ? I noticed image swaps don't work on faces that are too zoomed in, even if they work for video swaps and you can't specifiy an output filename, only the folder. If you do, no output file will be generated.

[mata3o]

i apologize for the long spam but its the only way i could think to show you what all is happening. I did all install instructions to the letter (twice, clean install the 2nd time) to make sure i didnt miss anything. Would you have a recommendation for picture sizes for swap and also could you tell me what command lines you use for single face single picture swap? Ive tried everything i can think of and all variations of picture face swap(single multi etc). video works like a charm though with no errors. I greatly appreciate your reply and any help or insight you might be able to give, cheers!

(simswap) C:\Users\user1\Desktop\Simswap Overlord\SimSwap-main>python test_wholeimage_swapsingle.py --isTrain false --use_mask --name people --Arc_path arcface_model/arcface_checkpoint.tar --pic_a_path ./demo_file/testpicA.jpg --pic_b_path ./demo_file/testpicB.jpg --output_path ./output/result_whole_swapsingle.jpg ------------ Options ------------- Arc_path: arcface_model/arcface_checkpoint.tar aspect_ratio: 1.0 batchSize: 8 checkpoints_dir: ./checkpoints cluster_path: features_clustered_010.npy data_type: 32 dataroot: ./datasets/cityscapes/ display_winsize: 512 engine: None export_onnx: None feat_num: 3 fineSize: 512 fp16: False gpu_ids: [0] how_many: 50 id_thres: 0.03 image_size: 224 input_nc: 3 instance_feat: False isTrain: False label_feat: False label_nc: 0 latent_size: 512 loadSize: 1024 load_features: False local_rank: 0 max_dataset_size: inf model: pix2pixHD multisepcific_dir: ./demo_file/multispecific nThreads: 2 n_blocks_global: 6 n_blocks_local: 3 n_clusters: 10 n_downsample_E: 4 n_downsample_global: 3 n_local_enhancers: 1 name: people nef: 16 netG: global ngf: 64 niter_fix_global: 0 no_flip: False no_instance: False no_simswaplogo: False norm: batch norm_G: spectralspadesyncbatch3x3 ntest: inf onnx: None output_nc: 3 output_path: ./output/result_whole_swapsingle.jpg phase: test pic_a_path: ./demo_file/testpicA.jpg pic_b_path: ./demo_file/testpicB.jpg pic_specific_path: ./crop_224/zrf.jpg resize_or_crop: scale_width results_dir: ./results/ semantic_nc: 3 serial_batches: False temp_path: ./temp_results tf_log: False use_dropout: False use_encoded_image: False use_mask: True verbose: False video_path: ./demo_file/multi_people_1080p.mp4 which_epoch: latest -------------- End ---------------- (142, 366, 4) C:\Users\user1\MiniConda3\envs\simswap\lib\site-packages\torch\serialization.py:656: SourceChangeWarning: source code of class 'torch.nn.parallel.data_parallel.DataParallel' has changed. Tried to save a patch, but couldn't create a writable file DataParallel.patch. Make sure it doesn't exist and your working directory is writable. warnings.warn(msg, SourceChangeWarning) C:\Users\user1\MiniConda3\envs\simswap\lib\site-packages\torch\serialization.py:656: SourceChangeWarning: source code of class 'torch.nn.modules.conv.Conv2d' has changed. Tried to save a patch, but couldn't create a writable file Conv2d.patch. Make sure it doesn't exist and your working directory is writable. warnings.warn(msg, SourceChangeWarning) C:\Users\user1\MiniConda3\envs\simswap\lib\site-packages\torch\serialization.py:656: SourceChangeWarning: source code of class 'torch.nn.modules.batchnorm.BatchNorm2d' has changed. Tried to save a patch, but couldn't create a writable file BatchNorm2d.patch. Make sure it doesn't exist and your working directory is writable. warnings.warn(msg, SourceChangeWarning) C:\Users\user1\MiniConda3\envs\simswap\lib\site-packages\torch\serialization.py:656: SourceChangeWarning: source code of class 'torch.nn.modules.activation.PReLU' has changed. Tried to save a patch, but couldn't create a writable file PReLU.patch. Make sure it doesn't exist and your working directory is writable. warnings.warn(msg, SourceChangeWarning) C:\Users\user1\MiniConda3\envs\simswap\lib\site-packages\torch\serialization.py:656: SourceChangeWarning: source code of class 'torch.nn.modules.pooling.MaxPool2d' has changed. Tried to save a patch, but couldn't create a writable file MaxPool2d.patch. Make sure it doesn't exist and your working directory is writable. warnings.warn(msg, SourceChangeWarning) C:\Users\user1\MiniConda3\envs\simswap\lib\site-packages\torch\serialization.py:656: SourceChangeWarning: source code of class 'torch.nn.modules.container.Sequential' has changed. Tried to save a patch, but couldn't create a writable file Sequential.patch. Make sure it doesn't exist and your working directory is writable. warnings.warn(msg, SourceChangeWarning) C:\Users\user1\MiniConda3\envs\simswap\lib\site-packages\torch\serialization.py:656: SourceChangeWarning: source code of class 'torch.nn.modules.pooling.AdaptiveAvgPool2d' has changed. Tried to save a patch, but couldn't create a writable file AdaptiveAvgPool2d.patch. Make sure it doesn't exist and your working directory is writable. warnings.warn(msg, SourceChangeWarning) C:\Users\user1\MiniConda3\envs\simswap\lib\site-packages\torch\serialization.py:656: SourceChangeWarning: source code of class 'torch.nn.modules.linear.Linear' has changed. Tried to save a patch, but couldn't create a writable file Linear.patch. Make sure it doesn't exist and your working directory is writable. warnings.warn(msg, SourceChangeWarning) C:\Users\user1\MiniConda3\envs\simswap\lib\site-packages\torch\serialization.py:656: SourceChangeWarning: source code of class 'torch.nn.modules.activation.Sigmoid' has changed. Tried to save a patch, but couldn't create a writable file Sigmoid.patch. Make sure it doesn't exist and your working directory is writable. warnings.warn(msg, SourceChangeWarning) C:\Users\user1\MiniConda3\envs\simswap\lib\site-packages\torch\serialization.py:656: SourceChangeWarning: source code of class 'torch.nn.modules.dropout.Dropout' has changed. Tried to save a patch, but couldn't create a writable file Dropout.patch. Make sure it doesn't exist and your working directory is writable. warnings.warn(msg, SourceChangeWarning) C:\Users\user1\MiniConda3\envs\simswap\lib\site-packages\torch\serialization.py:656: SourceChangeWarning: source code of class 'torch.nn.modules.batchnorm.BatchNorm1d' has changed. Tried to save a patch, but couldn't create a writable file BatchNorm1d.patch. Make sure it doesn't exist and your working directory is writable. warnings.warn(msg, SourceChangeWarning) input mean and std: 127.5 127.5 find model: ./insightface_func/models\antelope\glintr100.onnx recognition find model: ./insightface_func/models\antelope\scrfd_10g_bnkps.onnx detection set det-size: (640, 640) C:\Users\user1\MiniConda3\envs\simswap\lib\site-packages\torch\nn\functional.py:3500: UserWarning: The default behavior for interpolate/upsample with float scale_factor changed in 1.6.0 to align with other frameworks/libraries, and now uses scale_factor directly, instead of relying on the computed output size. If you wish to restore the old behavior, please set recompute_scale_factor=True. See the documentation of nn.Upsample for details. "The default behavior for interpolate/upsample with float scale_factor changed " C:\Users\user1\MiniConda3\envs\simswap\lib\site-packages\torch\nn\functional.py:3455: UserWarning: Default upsampling behavior when mode=bilinear is changed to align_corners=False since 0.4.0. Please specify align_corners=True if the old behavior is desired. See the documentation of nn.Upsample for details. "See the documentation of nn.Upsample for details.".format(mode)

**** Done ! ****

[dasasf]

I use the commands described here: https://github.com/neuralchen/SimSwap/blob/main/docs/guidance/usage.md . It looks to me your problem is that you specify an output file name.

Try running this command and see if it works:

python test_wholeimage_swapsingle.py --isTrain false --use_mask --name people --Arc_path arcface_model/arcface_checkpoint.tar --pic_a_path ./demo_file/testpicA.jpg --pic_b_path ./demo_file/testpicB.jpg --output_path ./output/

[mata3o]

THANK YOU SO MUCH, it does seem the problem was me putting in a file name (i was putting in result_swapsingle for a name) and the problem compiled by it being rather picky on image size for swapping pictures! Still throws all those error messages but it works so im totally fine with that. Again i very much appreciate you taking the time to lend a fresh perspective on it all! Cheers m8