search

neuralchen / SimSwap

An arbitrary face-swapping framework on images and videos with one single trained model!
Other
4.55k stars 895 forks source link

Not Issue: Image sequence and Alpha channel #79

Open Echolink50 opened 3 years ago

Echolink50 commented 3 years ago

Sorry if this is already been covered but it is a little bit difficult to find information. Trying to Google about Simswap brings up phone hack info. I was wondering about using image sequences instead of videos. I was mainly interested in getting Alpha Channel from png's or something like that. Does Simswap support a video format that can use Alpha Channel like QT RLE? I see that there is a new mask function but I didn't find a lot of information about it. Can Simswap get Alpha channel in and put out a format with Alpha Channel? I would like to put in just the faces with an alpha background and get out just the swapped faces with an alpha background. I also prefer to work with image sequences over video if it's possible. Thanks everyone.

woctezuma commented 3 years ago

When it comes to SimSwap, videos are image sequences with audio, so working with image sequences is doable.

As for the transparent channel, I don't think that it is possible, because SimSwap assumes that the images are RGB, and not RGBA.

https://github.com/neuralchen/SimSwap/blob/1b8f48279570dda55aa0843987d9a7b6ee87af81/test_one_image.py#L41

Maybe it would be possible to embed SimSwap in a personal framework in which you keep track of the transparency with a mask, convert images from RGBA to RGB, run SimSwap, and then apply your mask to get the transparency back on the output.

You may face an issue at the borders of the face, due to the blending though: after conversion from RGBA to RGB, transparency will be understood as background color (black or white, I don't know), so the blending will mix some of this background color with the face.

https://github.com/neuralchen/SimSwap/blob/1b8f48279570dda55aa0843987d9a7b6ee87af81/util/reverse2original.py#L136-L140

https://github.com/neuralchen/SimSwap/blob/1b8f48279570dda55aa0843987d9a7b6ee87af81/util/reverse2original.py#L169

Echolink50 commented 3 years ago

Ok thank you for the quick reply. What is the new mask feature mentioned

"2021-07-19: Obvious border abruptness has been resolved. We add the ability to using mask and upgrade the old algorithm for better visual effect"

Is this like i can provide a mask video to help simswap to only see the face? Or is it something else. How do i use it? Thanks

woctezuma commented 3 years ago

Is this like i can provide a mask video to help simswap to only see the face? Or is it something else. How do i use it? Thanks

I think it is using face segmentation (rather than just face detection). cf. https://github.com/neuralchen/SimSwap/issues/59 for instance

Echolink50 commented 3 years ago

Ok. Thank you very much