How to obtain the FLAME params for custom input?

[ankit-gahlawat-007]

Hello, first of all, thanks a lot for the wonderful work and making it open source.

I wanted to run this on a custom dataset. I am using the colmap data for camera params but that would not include the mesh information. Could you please point out to how could I get the FLAME params?

For example, for the dataset 306, 00000.npz has these shape values: translation.npy (1,3) rotation.npy (1,3) neck_pose.npy (1,3) jaw_pose.npy (1,3) eyes_pose.npy (1,6) shape.npy (300) expr.npy (1,100) static_offset.npy (1,5143,3) If there is some method, which could give all these values for the custom input frames, it would be really helpful.

Thanks in advance!

[nlml]

Hi @ankit-gahlawat-007

There are a number of ways to estimate FLAME parameters.

You can do offline FLAME tracking, using a method like https://github.com/Zielon/metrical-tracker

Or you could use a method like EMOCA to estimate these in real time https://github.com/radekd91/emoca/

This will work for trying to animate an existing avatar. If you are trying to train a new avatar, it will be a lot more difficult as none of these methods give you static_offset. But you could try training without these.

[ankit-gahlawat-007]

Thanks a lot Liam for the references, will check these out.

[starc52]

@ankit-gahlawat-007 Did you figure out how to get the right format for the FLAME tracking output?

[ankit-gahlawat-007]

Hi @starc52, I have got the outputs from the metrical-tracker. Apart from static_offset, it also seems to be missing the neck pose values. Although the rotation and translation values are present in the camera params, I am not sure how will they correspond to the actual FLAME head params. I am yet to make the final code changes to accommodate the metrical-tracker output inside GaussianAvatars though.

[starc52]

Hi @ankit-gahlawat-007 I think this should help. rotation.npy should mean global rotation, while neck_pose should mean head rotation around the neck. Check this out: https://github.com/TimoBolkart/TF_FLAME/issues/40#issuecomment-739552785

[jeb0813]

Hi @nlml , i got FLAME params from EMOCA, but the shape is 100, 6, 50 for shape, pose, expression.

The author explained

The 6D pose corresponds to global rotation and jaw rotation (parameters 1-3 and 7-9 of the 18-dimensional FLAME pose vector). The 100-dimensional identity shape and the 50-dimensional expression shape are the first 100 / 50 parameters of the FLAME identity shape / expression shape vectors. The remaining 200 FLAME identity and 50 expression remain at zero.

Should I just ignore other params and simply put them to ZERO? Will that work?

[nlml]

@jeb0813 Yes you can set everything else to zero, since it is a PCA basis the shape and expression dimensions are of decreasing importance.

[ShiweiJin]

Hi @starc52, I have got the outputs from the metrical-tracker. Apart from static_offset, it also seems to be missing the neck pose values. Although the rotation and translation values are present in the camera params, I am not sure how will they correspond to the actual FLAME head params. I am yet to make the final code changes to accommodate the metrical-tracker output inside GaussianAvatars though.

Hi @ankit-gahlawat-007, thank you for your sharing. Can I ask for more hints about the neck_pose? I also used the metrical-tracker but still cannot get the neck_pose.

• I assume the neck_pose as the head pose. Please correct me if this assumption is wrong.
• Did I implement correctly if I cannot get the neck pose from the metrical-tracker?
• Let assume we don't have the neck_pose from the metrical tracker. Then would the estimated camera extrinsic matrix vary if the subject shows different head poses in two videos recorded by a fixed camera?

Thank you very much for any hints.

[XvHaidong]

Hi @ankit-gahlawat-007

There are a number of ways to estimate FLAME parameters.

You can do offline FLAME tracking, using a method like https://github.com/Zielon/metrical-tracker

Or you could use a method like EMOCA to estimate these in real time https://github.com/radekd91/emoca/

This will work for trying to animate an existing avatar. If you are trying to train a new avatar, it will be a lot more difficult as none of these methods give you static_offset. But you could try training without these.

Hi @nlml, I noticed that the dimensions of "eyes" and "jaws" in "metrical-tracker" are 12 and 6, respectively, whereas in your paper, they are 6 and 3. Could you advise on how I can adjust these dimensions to customize my dataset?

[jryebread]

hi @nlml possible to share a script for using metrical tracker or similar to get the params needed for custom input?

If so would appreciate anyone sharing script to convert custom frames/mp4 video to get the necessary npz file and what is needed, I was surprised this wasn't provided as part of the instructions, spent a few hours setting up the env only to find I am stuck on this step.

[jryebread]

hi @XvHaidong did you figure out how to run custom input?