search

TMElyralab / MusePose

MusePose: a Pose-Driven Image-to-Video Framework for Virtual Human Generation
Other
2.29k stars 167 forks source link

How to lip sync? #68

Closed mrtpk closed 3 months ago

mrtpk commented 3 months ago

Hi Team,

Great work. Thank you for sharing.

Once I created a video using MusePose, is it possible to create a lip sync video using MuseTalk? Can you please guide me on this? Thanks.

Best, Thomas.

czk32611 commented 3 months ago

You can just input the video to MuseTalk according to here

mrtpk commented 3 months ago

Thanks for your suggestion; I ran MusePose first with an image and a reference video. I then extracted the audio from the reference video. This audio and output from MusePose were given as input through a config file to MuseTalk.

Is the following workflow okay for enhancing the face and lip sync? Your opinion on the following workflow will be very helpful.

workflow 1: 1) Crop the face part and generate lip sync for that part using MuseTalk 2) Use MusePose to generate the movement 3) Use FaceFusion to fuse the output from MosePose and MuseTalk

or

workflow 2: 1) Use MusePose to generate a pose video 2) Use MuseTalk to generate lip sync on MusePose output 3) Final step, use FaceFusion to enhance the face

I appreciate your help on this and is looking forward to hearing your thoughts.