search

TimoBolkart / voca

This codebase demonstrates how to synthesize realistic 3D character animations given an arbitrary speech signal and a static character mesh.
https://voca.is.tue.mpg.de/en
1.15k stars 273 forks source link

Fitting a TurboSquid Head mesh to a FLAME topology #43

Closed mbdash closed 3 years ago

mbdash commented 4 years ago

Hi, thank you for this great library.

From my understanding, the library only allow to use 3D models with FLAME topology, but in the demo video, a model (Churchill) from TurboSquid was used.

I cannot find clear instruction on how to use a "hand made" 3D model with this library. I tried using Blender to convert an .obj head mesh to .ply, but obviously, the topology is different and it failed.

I see I could maybe use RingNet to generate a mesh from an image, but what if I have a 3D model already?

TLDR Could you please elaborate on the procedure to take a regular .obj and use it with this library.

Thank you very much for your time and any information you can share.

TimoBolkart commented 4 years ago

Hi, there are two steps you need to do, First, you need to register the scan with the FLAME model (for details you please see the paper). Then you need to undo the effects of pose (i.e. global orientation, head pose, jaw pose) and facial expression. After this process, the mesh is in zero pose (i.e. canonical FLAME pose). Then it can be used together with VOCA.

mbdash commented 4 years ago

Hi thank you for taking the time to answer.

But it still doesn't clarify how to take a non-FLAME mesh and convert it to FLAME topology.

You mention: First, you need to register the scan with the FLAME model My question relates to using a mesh which is not a scan but a 3D mesh modeled by a 3d artist. In the paper, I can read: the authors provided us with a static mesh, to which we aligned the FLAME topology. But how can you align the FLAME topology over a 3D model?

You also mention: Then you need to undo the effects of pose A handmade 3D model should already by in a neutral pose, from my understanding it just need to have it's transform reset so that the mesh position overlays one of the FLAME ply meshes. I think I can do it using Blender.

(fyi, my skill set is more about 3d animation then ML.)

mbdash commented 4 years ago

Has anyone got any hints regarding how to register a 3D mesh, modeled by a 3d artist, to FLAME topology? IE a mesh purchased on TurboSquid. I am sorry, I fail to find instructions in the paper.

TimoBolkart commented 4 years ago

Thank you, please check out Section 4.2 of the FLAME paper on how to align a general scan that is not in FLAME topology. The basic idea is to replace the vertex to vertex distance from the fit_3D_mesh demo with a general point to surface distance and minimize this instead. This is often referred to non-rigid ICP.

Another option could be using the provided FLAME Blender plugin to output a mesh in FLAME topology that then can be animated by VOCA.

TimoBolkart commented 3 years ago

We also just added some demo code to fit FLAME to a scan of arbitrary topology in the flame-fitting repo.