search

ai-forever / ghost

A new one shot face swap approach for image and video domains
Apache License 2.0
1.25k stars 265 forks source link

Video: face jitters #14

Open bmc84 opened 2 years ago

bmc84 commented 2 years ago

Hi,

I've tested the video swap functionality using the provided Colab, and in every test, the face "jitters". It looks like the scale of the face is changing by a very small amount. Sometimes the eyebrows are not aligned, either. So we have two copies of the eyebrows (in the instance that the original video raises their eyebrows).

Also, the mouth & eyes do not correctly match the source.

This video shows an example of the jittery face & eyes not pointing correctly.

https://user-images.githubusercontent.com/87626308/151474362-137800dd-d38e-4180-99a2-20f9f546363d.mp4

And this shows how the mouth should look in the source, how SimSwap correctly translates it, and how sber-swap does not have the correct eyes or mouth.

mouth

Are these issues with the approach, or would training the model for longer resolve the issues?

AlexanderGroshev commented 2 years ago

@bmc84 Thanks for your feedback!

Jittering of the face is a challenge we are working on. Longer training may partially solve this task.

Two copies of the eyebrows may appear when the eyebrows in the original video are raised too high. This issue can be solved at the stage of blending output of the model back into the original video by expanding the face mask (check masks.py).

Speaking about swapping face attributes such as mouth and eyes, the priority task is to make them visually more like source photo attributes, thus they may be less similar to original video.

Can you please share the source photo and video so we can analyze your example in more detail.

alcanunsal commented 2 years ago

Hi @AlexanderGroshev , firstly, thanks for the amazing work! Could you elaborate a little more on the eyebrow part? I read issue #22 but didn't get significant improvement in 2 copies of eyebrows showing at the same time issue. I am wondering if I did something wrong or if it just doesn't resolve the issue. Could you give some specifics about the values we could give to eyebrows_expand_mod & erode_and_blur variables and how they should affect the result? Thanks in advance

hippotabek commented 11 months ago

Hey! thank you for sharing your work.

Are you planning to solve face jitter issue?