Masking animated Head

[instant-high]

Sorry, but have to ask again.

I've made major modifications to your code so I can use it for fullframe face/head/partswap.

I use the openCV caffemodel for head tracking and cropping (expanded and squared bbox) for animating the image. Then the animated image is masked and put back to the original fullframe video, including a automatic color correction.

For masking the full head or just parts of it I use this (modified) face-makeup repo (https://github.com/zllrunning/face-makeup.PyTorch)

I must admit I don't understand all of your code in full. So I wonder if you use masking in one of the modules and if it would be possible to get that mask...?

Without the need to use the face-makeup parsing the faceswap code could run about 30 percent faster.

Regards Thomas

[asrlhhh]

Hey Thomas, that sounds so cool. I have the plan to do the same thing. To answer you question, from their paper, I don't think they have mask in place but just motion and keypoints predictions. I also have to crop the head, use face alignment model to align the heads to achieve optimal animation results. Putting the animated face back to the original frame will be my last step. Do you mind creating a PR for that? I think many ppl will be grateful for it, as currently the good animation effect can only be achieved when the driving and reference images are perfectly aligned

[instant-high]

I can make a package for download including all neccessary files. Maybe someone can optimitze my code. (I'm new to python, coming from 35 years Basic and a little bit c++) I just use a plain text editor for now. My script has a lot of other features like rescaling input, cropping output, audio or not, start and end frame (cutting the video) and some functions during animation by pressing some keys, like padding, colourcorrect, manual alignment.....

[asrlhhh]

That's awesome. I have 6+ years of Python experience on CVML and can definitely help refine the codes. One interest phenomenon I found is that it might be easier to crop the driving video instead of reference image. I think overall the model needs the keypoints (be it face landmarks, body landmarks, or houses etc) to be aligned on scale. I was thinking that you can upload on your github repo so that when I or others make improvement, it's easier for us to recognize your contribution. But if you want to keep it private, my email is: asrielhan@gmail.com

[mariolew]

@instant-high Hi, can I get your script? I'm really interested in fullframe face/head/partswap.

[Zhang-Xiaoyi]

I can make a package for download including all neccessary files. Maybe someone can optimitze my code. (I'm new to python, coming from 35 years Basic and a little bit c++) I just use a plain text editor for now. My script has a lot of other features like rescaling input, cropping output, audio or not, start and end frame (cutting the video) and some functions during animation by pressing some keys, like padding, colourcorrect, manual alignment.....

Hi @instant-high, Can I get a copy of your script? I am very interested in the face swap part. zhangxiaoyi1127@gmail.com

[instant-high]

It's still a work in progress. I switched to insightface/onnx for detection and alignment. Code is really chaotic. But I can make a package with everything additionally needed.

[instant-high]

@Zhang-Xiaoyi sent you the script.

Here you can find some older version examples of headswap/partswap https://www.youtube.com/@heinzelmann1963

[andrewkuo]

@instant-high Can I get a copy of your script as well? I would love to try it out.

[instant-high]

@andrewkuo I've sent you a link to a package including all files including checkpoints for face_parsing and insightface. Hopefully it's complete this time and you get it working (anaconda win10 env)

[Chromer163]

mark

[makingpippop]

@instant-high I also would be interested to check out what you got!

[Hujiazeng]

Hi, @instant-high, I would love to try it out. Can I get a copy of your script?