search

Inferencer / LipSick

🤢 LipSick: Fast, High Quality, Low Resource Lipsync Tool 🤮
The Unlicense
185 stars 28 forks source link

Could you check if I'm implementing it correctly? #22

Open ksw29zz opened 6 months ago

ksw29zz commented 6 months ago

Thank you always for your amazing work! :)

I have a few questions, so I wanted to talk to you about them.

  1. The mouth seems to have a low resolution. I wonder if this is something I need to train to improve.
  2. When small and large faces appear alternately in the video, it looks like the video is glitching. I wonder if I'm doing something wrong here.
  3. I have attached a video, and in this particular video, it can't seem to detect the face. Could you check it out for me? video_and_audio.zip
Inferencer commented 6 months ago

  1. I will add up-scaling of reference frames as an option and make conversions and encodings slower to increase quality I could train to double the resolution but that's a long way away. To be clear though this tool works phenomenally with the correct source footage as it was trained on studio lighting with static cameras so news readers are what works best but if you search for celebs reading mean tweets you will find some good vids as the lighting is great, the background is blue and the resolution of the teeth is good due to the high quality recording. however they do look down at their laptops a lot so will be best to clip them so the face is looking at the camera at all times.

  2. I recently changed the crop radius to be consistent so when the face is overlapped onto the original video there is no flickering but this can cause issues if the face is moving back and fourth or the camera is too much, in your circumstance of small and large faces appearing alternately n the source footage.. I will implement a scene change detection so we can use different crop radius's for each scene as well as different custom ref frames.

  3. The vid you shared appears to be ok in terms of the face in each frame however I wonder if the fps is the cause of the issue as when it converts to 25fps it maybe doing something it shouldn't so I will investigate that. Looking for a good source vid should take up a longer time than inference so in the case of Taylor I would have used https://www.youtube.com/watch?v=XnbCSboujF4 with the time stamps of 6:47 to 6:51