How do the blendshapes work?

[samsgates]

Hi Patrik,

I have added blendshape (expression_blendshapes_3448.bin) to PCA model using the below code. i have read that output OBJ file in text editor, and there is no additional vertex appears in obj file. line count is same (13634). But i can see some difference in model (mouth opened slightly), how to identify blendshapes are added in model, and how to test ?

find the attached output OBJ file out.obj.zip

vector<float> fitted_coeffs = fitting::fit_shape_to_landmarks_linear(morphable_model, affine_from_ortho, image_points, vertex_indices);

vector<morphablemodel::Blendshape> blendshapes = morphablemodel::load_blendshapes(blendshapesfile.string());
vector<float> shape_coefficients, blendshape_coefficients;
Mat shape_instance = fitting::fit_shape_model(affine_from_ortho, morphable_model, blendshapes, image_points, vertex_indices, 10.0f, shape_coefficients, blendshape_coefficients);

render::Mesh mesh = morphable_model.draw_sample(fitted_coeffs, vector<float>());

auto merged_shape = morphable_model.get_shape_model().draw_sample(fitted_coeffs) + to_matrix(blendshapes) * Mat(blendshape_coefficients);

render::Mesh merged_mesh = morphablemodel::detail::sample_to_mesh(merged_shape, morphable_model.get_color_model().get_mean(), morphable_model.get_shape_model().get_triangle_list(), morphable_model.get_color_model().get_triangle_list(), morphable_model.get_texture_coordinates());

// Save the mesh as textured obj:
outputfile += fs::path(".obj");
render::write_textured_obj(merged_mesh, outputfile.string());

[patrikhuber]

Hi Sam,

Blendshapes do not add vertices to the model. They're offsets, one offset value for each vertex.

To kind of "see" what's going on, you could inspect the values of blendshape_coefficients. The line

auto merged_shape = morphable_model.get_shape_model().draw_sample(fitted_coeffs) + to_matrix(blendshapes) * Mat(blendshape_coefficients);

then creates a shape consisting of the PCA model with added expressions. This is equivalent to formula (6) in our most recent report.

[samsgates]

That means any one expression (blendshape) will begenerated in output. is it correct? so not all blendshapes are included in model. how to specify expression (blendshape) name (like "angry") to generate with output? so that i can get only angry face expression in output.

[patrikhuber]

The vector blendshape_coefficients holds the amount of each coefficient that is added. The variable blendshapes contains the 6 blendshapes of type morphablemodel::Blendshape, and as you can see in the code and documentation, they include the name of the respective expression.

[patrikhuber]

I'm closing this. Feel free to re-open if there are still questions regarding the blendshapes.