search

yerfor / GeneFace

GeneFace: Generalized and High-Fidelity 3D Talking Face Synthesis; ICLR 2023; Official code
MIT License
2.43k stars 290 forks source link

Head and Torso Separation #241

Closed kavita-gsphk closed 4 months ago

kavita-gsphk commented 7 months ago

I have trained Geneface on a new person. I have followed all the instructions given in the docs folder. I used RAD-NeRF-based rendering. But in the output, the head and torso have a separation. How can I improve it? I am attaching a cropped video for you to look over.

I really appreciate any help you can provide.

https://github.com/yerfor/GeneFace/assets/63964682/856c5474-7ce4-48f4-af66-c0ea213fd71f

yerfor commented 7 months ago

Hi, have your train the torso nerf? If so, setting torso_head_aware: true in the config file may help. Please refer to egs/datasets/videos/May/lm3d_radnerf_torso_head_aware.yaml

kavita-gsphk commented 7 months ago

Thank you for your response.  Yes, I have trained the torso. The above result is generated by setting torso_head_aware: true only. So, I have noticed results generated with lm3d_radnerf_torso_head_aware.yamland lm3d_radnerf_torso.yaml didn't had any noticeable difference. 

yerfor commented 7 months ago

I have watched the provided video, and it seems that the face parsing wrongly regards some of the neck part as the head part. As a result, the HeadNeRF renders the head along with the neck. which causes the artifacts in the neck part as shown in the video (the upper neck is rotating along with the head). Maybe you could try to improve the face parsing accuracy of the training video. Here are two suggestions:

(1) Resort to more advanced face parsing methods, such as mediapipe's serfie API. (2) Handcraft the segmentation map, which is time consuming though.

kavita-gsphk commented 7 months ago

Ok, Let me check. Thank you so much!!