search

zhangzc21 / DynTet

178 stars 21 forks source link

Eyes will not blink, face segmentation is not good, how to optimize? #4

Closed einsqing closed 3 months ago

einsqing commented 3 months ago

Eyes will not blink, face segmentation is not good, how to optimize?

zhangzc21 commented 3 months ago

This is very strange. Could you provide me with more information? If the code works well on the Obama video and outputs reasonable results?

zhangzc21 commented 3 months ago

Note both the eyeblink and mouth movements in DynTet are driven by 3DMM coefficients. The file "obama_sing_sadtalker.npy" is generated by SadTalker. We employ the first frame in the Obama video as the driven image, while not providing the eyeblink reference. Thus the 3DMM coefficients might not drive the eyeblink well. I think you can try SadTalker to generate 3DMM coefficients, accordingly providing eyeblink reference. It is promising to produce better results.

By the way, we follow previous works to achieve face segmentation by a pre-trained model. I think it is hard to optimize. A better way to enhance the visual effect might be to crop the face area as tightly as possible and paste it into the original image.

einsqing commented 3 months ago

How do I generate a 3dmm? face_recon3dmm.npy?or do I use this method? https://github.com/OpenTalker/SadTalker/blob/cd4c0465ae0b54a6f85af57f5c65fec9fe23e7f8/src/utils/preprocess.py#L46

hereTac commented 2 months ago

"A better way to enhance the visual effect might be to crop the face area as tightly as possible and paste it into the original image."

How can we interpret this? Does it mean that videos with larger head sizes are easier to crop and produce better results? @zhangzc21