search

yerfor / GeneFacePlusPlus

GeneFace++: Generalized and Stable Real-Time 3D Talking Face Generation; Official Code
MIT License
1.39k stars 203 forks source link

Mismatch of the head movement (raw video vs. generated video) #175

Open schxnhxlz opened 3 months ago

schxnhxlz commented 3 months ago

Hey there,

first of all great project! Had a lot fun playing around with it. My next step is to put the generated video back on top of the raw video.

videoonraw

To remain the real eye and forehead movement i want to mask out just the nose mouth and jaw:

Screenshot 2024-04-17 at 07 09 31

Unfortunateley the movement of the raw head and the "new" head doesnt match 100%. There is still a little mismatch. It looks like the head is wobbling.

https://github.com/yerfor/GeneFacePlusPlus/assets/58291636/c773ca8e-cbb3-4883-944c-0a504c70150f

I already set the hparams['infer_smooth_camera_path_kernel_size'] = 7 to 0 and isabled the smooth featues

 smo_euler = smooth_features_xd(batch['euler'])
 smo_trans = smooth_features_xd(batch['trans'])

setting drv_pose to static didnt help. Are there any other parameters I can try to edit to achieve a perfect match of the landmarks? Especially for the nose and eyes?

Thanks in advance!

jryebread commented 3 months ago

i have the same problem, hoping someone can answer this!

hlevring commented 1 month ago

I have in not looked closer at the repo, but was thinking that some of the comments in thread below may be related to this issue https://github.com/yerfor/GeneFacePlusPlus/issues/104