search

yfeng95 / DECA

DECA: Detailed Expression Capture and Animation (SIGGRAPH 2021)
Other
2.06k stars 418 forks source link

Question about 3D reconstruction from video #165

Open HowieMa opened 1 year ago

HowieMa commented 1 year ago

Hi, thanks for this fantastic work! When reconstructing the 3D face model from a video of a subject, it seems that the current demo code reconstructs the 3D face frame by frame, as in this code.
However, in practice, I find that this implementation cannot guarantee the consistency of shape code (identity code) across different frames. Thus, I'd like to know if is there any way that can maintain the shape consistency during the inference time as well, just like the shape consistency loss in the training time. Look forward to your reply. Thanks!

emlcpfx commented 1 year ago

Howie, I'm wondering the same thing. Did you ever make any progress with this?

HowieMa commented 1 year ago

Howie, I'm wondering the same thing. Did you ever make any progress with this?

Hi, I don't think the vanilla DECA can support this function. Maybe we can use DECA to get initial parameters and then apply an optimization-based method to refine the performance on a video, like this https://github.com/philgras/video-head-tracker

emlcpfx commented 12 months ago

They’re using DECA in this— https://imaging.cs.cmu.edu/local_deformation_fields/

I asked them if they had modified DECA at all to get better tracking, and they said, “ We finetuned the results by projecting the tracked 3d landmarks from DECA to 2d landmarks detected from a standard landmark detector.”

Echo-jyt commented 1 month ago

Hi, Did you find a good solution?