How to generate full head model?

[samsgates]

Hi patrikhuber, How to generate full head model from face image? how to create my own PCA model for input to fit-model tool.

Regards, Sam

[patrikhuber]

Hi Sam, You need 3D data (scans) to build your own model. Once the data is registered (in dense correspondence), you just apply PCA to that data. For example using cv::PCA, EIgen, or Matlab. You then simply store the mean, eigenvalues and eigenvectors as cv::Mat so they are compatible with eos' PcaModel.

[samsgates]

So, if i use full head model as PCA, output will generate full head model. is that correct? can you please provide any sample 3D data with full head model. BlendShape : How to apply blendshape after getting output mesh?

[patrikhuber]

So, if i use full head model as PCA, output will generate full head model. is that correct?

Yes, if your input data is full 3D head scans, the PCA is of course over the full head.

can you please provide any sample 3D data with full head model.

I'm sorry, we do not have sample full head 3D scans.

BlendShape : How to apply blendshape after getting output mesh?

They're added linearly to the PCA model, so a shape is S = PcaModel + B * psis where B is the blendshape matrix and psis the blendshape coefficients. In the code it's exactly that formula:

auto shape = morphable_model.get_shape_model().draw_sample(shape_coefficients) + to_matrix(blendshapes) * Mat(blendshape_coefficients);
render::Mesh mesh = morphablemodel::detail::sample_to_mesh(shape, morphable_model.get_color_model().get_mean(), morphable_model.get_shape_model().get_triangle_list(), morphable_model.get_color_model().get_triangle_list(), morphable_model.get_texture_coordinates());

See here.

[samsgates]

got it..! thanks for your support

[patrikhuber]

Great! You're welcome.

[ShaojunBian]

@patrikhuber

Hi Patrikhuber,

As you explained about the blendshape , can I ask the blendshape are only the expression blendshapes of the meanshape?

or need create more blendshapes for different peoples' face in the scans dataset?

Thank you so much for your time.

*They're added linearly to the PCA model, so a shape is S = PcaModel + B psis where B is the blendshape matrix and psis the blendshape coefficients. In the code it's exactly that formula:**

[patrikhuber]

Hi @ShaojunBian,

Your post is a bit hard to read, would be nice if the formatting was better. This is also a bit off-topic here. The blendshapes are generic blendshapes (not subject specific, no PCA). You can find more details about it in my PhD thesis, Section 3.3 B).

[ShaojunBian]

Hi @patrikhuber,

Thank you very much for your so quick answer! It helps me a lot.

What I asked is if I would like to prepare my own morphable model.

(1) for example, use 60 different face scans in neutral pose to create the PCA model.

(2) the next step is to prepare the blendshape data, should I just create six expressions on the mean face? or do I need to create six expression blendshapes for each of the 60 different face, it means totally 60*6=360 blendshapes.

From your thesis, I understand that I only need to prepare 6 expressions for the mean face, is that correct?

Thank you again for your time and your brilliant work.

[patrikhuber]

Yes correct, for the 6 SFM blendshapes we just averaged the scans for each expression to get some "generic" expression blendshapes. Of course you could model them by hand as well. Or, as you correctly noted, if you have expressions for each of the faces, you can create identity-specific blendshapes or mix them all together in a PCA expression model.

Which way to do it depends on the data that you've got and your use case.

[ShaojunBian]

@patrikhuber , Thank you so much for your clear guidance.