How to convert the vertex into FLAME mode?

[lucasjinreal]

How to convert the vertex into FLAME mode?

[abdallahdib]

Hi adding support for FLAME is in my todo list (check readme). I am planning to contact FLAME authors to help with this. the important part is to get the albedo model transferred to FLAME. I will flag this this issue so that we may get some help with this.

[lucasjinreal]

@abdallahdib thank u! Really hoping for this feature! I am not sure if mapping the idx from NextFace vertex to FLAME accordingly semantic idx can work or not, but seems there should a very common way achademic using between this conversion.

Becuase if we want use it into industrial, a cropped front face just doesn't enough.

BTW, since the nextface generates a high fedelity face reconstruction, so here I wanna think further, does it possible make a expression basis of it? so that we can driven the face using face mocap data.

[abdallahdib]

U can transfer the 3D model of NextFace to other topologies like FLAME, u need to align the two templates to find the correspondances between the two meshes. I have never done this but there is some tools that can do this for u. The model used by NextFace (basel 2017) comes with its own expression basis. Driving the model from mocap data needs to transfer the mocap expression to basel one. The main difficulty i see here is that basel 2019 uses expression basis while other format use blendshape vectors.

[abdallahdib]

I came across this from the original authors of FLAME: https://github.com/TimoBolkart/BFM_to_FLAME

[lucasjinreal]

@abdallahdib I have to admit I have once tried this lib, turns out their dependeices like a nightmare.... be careful about it.

[lucasjinreal]

@abdallahdib You just reminded me. blendshapes actually more comn used, even thought I think they are just same thing. Does it possible we directly using Apples's 52 model to train the model rather than using BFM? Apple have released there face model, it was simple like FLAME.

The reason why using Apple's 52 face based, it's that iPhones ARKit are mostly commonly used to capture face motion data now, and seemlessly intergated into Unrealengine, Maya etc. The models as well.

[abdallahdib]

Hi, how does the apple template model looks like ? is it a parameteric face model? NextFace optimizes the coefficients of the parameteric face model (identity, expression and albedo). If apple face model follow the same then there should be a way to optimize its parameters. We still need to convert the albedo model used by basel 2009 (from https://github.com/waps101/AlbedoMM/tree/master/scala ) to apple topology

[lucasjinreal]

They have 52 blendshapes and according expression definition:

does these can be learned?

[abdallahdib]

so i think it is possible to directly optimize these blendshapes, we need to transfer first these delta blendshapes to basel topology. transfering requires to register each blendshape vector to the template mesh of basel. there is some tools that can do this transfer.

[spicer-c]

I came across this from the original authors of FLAME: https://github.com/TimoBolkart/BFM_to_FLAME

This at the moment requires using model2017-1_bfm_nomouth.h5 instead of model2017-1_face12_nomouth.h5, if this used the bfm model instead of just face you could take the obj generated from here and feed it into this. Not sure if that would be possible or not. I don't know how the normals and uv pickle files were generated so can't test this myself

[abdallahdib]

the normals hold the neighbors vertices for fast normal calculation at runtime, u can replace it with a classical normal computation (redner has functions for normals calculation). For the uv pickle it holds the barycentric coordinates used to generate texture maps.

I didnt have time to check the BFM_to_FLAME code, but what nextface needs to work with flame is the statistical albedos retargeted to flame model. In practice u need to create a new morphablemodel class where u load the identity shape pca of flame, expression and the albedo model (for the albedo as mentioned u need to transfer the eigen vectors of albedo pca to flame).

[spicer-c]

Ah yes that makes sense as a way to directly use FLAME here.

For the uv pickle it holds the barycentric coordinates used to generate texture maps.

How did you generate these?

[abdallahdib]

yeah u can directly use FLAME as a new morphable model.
in fact it turns out that (just discovered it now) that the albedo model of smith et al 20, is already transfered to FLAME check it from here: https://github.com/waps101/AlbedoMM/releases/tag/v1.0 (albedoModel2020_FLAME_albedoPart.npz)

I think FLAME has its barycentric coordinates (need to check on FLAME side)