Question about fitting with dlib landmarks

[virtualenv]

Dear Patrik,

I really would like your help here. I am trying to fit a image file to a mesh. I saw the 3 examples that you have, but it seems that none of them is doing what I would like. After more than 2 weeks trying, I decided to ask for help

Please look at this mesh, it is perfect (front) :

and perfect as well (side) :

this is the image of Mr. Morgan Freeman

And this is his mesh texturized :

So with Mr. Morgan Freeman, it all works perfectly. But other face image different then him, the isomap, does not fill well at all.

Here is a friend of mine :

Here is his mesh texturized :

Another example :

And his mesh :

Could you please help me understand why the isomap doesn't fit the mesh ?

Here is the code : using namespace eos; using namespace std; using Eigen::Vector2f; using Eigen::Vector4f;

dlib::cv_image<dlib::bgr_pixel> img(faceImage);
std::vector<dlib::rectangle> face = detector(img);

eos::core::LandmarkCollection<Eigen::Vector2f> landmarks;
int ibugId = 1;

landmarks.reserve(68);
if (face.size() > 0)
{
    dlib::full_object_detection facemarks = predictor(img, face[0]);
    //drawLandmarks(faceImage, landmarks);
    for (size_t i = 0; i < facemarks.num_parts(); i++)
    {
        eos::core::Landmark<Eigen::Vector2f> landmark;
        landmark.name = to_str(ibugId);
        landmark.coordinates[0] = facemarks.part(i).x();
        landmark.coordinates[1] = facemarks.part(i).y();
        landmark.coordinates[0] -= 1.0f;
        landmark.coordinates[1] -= 1.0f;
        landmarks.emplace_back(landmark);
        ++ibugId;
    }
}
cv::Mat image = faceImage.clone();

// Fit the 3DMM:
fitting::RenderingParameters rendering_params;
vector<float> shape_coefficients, blendshape_coefficients;
vector<Eigen::Vector2f> image_points;
vector<Eigen::Vector4f> model_points;
vector<int> vertex_indices; 
core::Mesh mesh;

for (int i = 0; i < landmarks.size(); ++i)
{
    auto converted_name = landmark_mapper.convert(landmarks[i].name);
    if (!converted_name)
    { // no mapping defined for the current landmark
        continue;
    }
    int vertex_idx = std::stoi(converted_name.value());
    auto vertex = morphable_model.get_shape_model().get_mean_at_point(vertex_idx);
    model_points.emplace_back(Vector4f(vertex.x(), vertex.y(), vertex.z(), 1.0f));
    vertex_indices.emplace_back(vertex_idx);
    image_points.emplace_back(landmarks[i].coordinates);
}
   eos::fitting::fit_shape_to_landmarks_linear(morphable_model.get_shape_model(), affine_cam, 
   image_points, vertex_indices);

   std::tie(mesh, rendering_params) = fitting::fit_shape_and_pose(morphable_model_with_expressions, landmarks, landmark_mapper, image.cols, image.rows, edge_topology, ibug_contour, model_contour, 1, eos::cpp17::nullopt, 30.0f);
  const Eigen::Matrix<float, 3, 4> affine_from_ortho = 
 fitting::get_3x4_affine_camera_matrix(rendering_params, image.cols, image.rows);
 const vector<float> fitted_coeffs = fitting::fit_shape_to_landmarks_linear(morphable_model.get_shape_model(), affine_from_ortho, image_points, vertex_indices);
mesh = morphable_model.draw_sample(fitted_coeffs, vector<float>());
const core::Image4u isomap = render::extract_texture(mesh, affine_from_ortho, core::from_mat(image));

Thank you so much!

[patrikhuber]

Hi,

I would advise you to check/visualise your landmarks, to make sure they're accurate. Also, fit_shape_and_pose returns you a mesh, and you're then subsequently overwriting that with another mesh estimated by the coefficients from fit_shape_to_landmarks_linear (which, for example, doesn't do contour fitting) - you probably don't want to do that. Indeed it looks in your images like the outer face landmarks ("contour" landmarks) might not be used.

I'm closing this as it is rather a user question than an issue with the library, but feel free to report back.

[virtualenv]

Dear Patrik,

First of all, thank you for such a quick reply. I am really sorry for the trouble, I know that this is not any issue with the library, just didn't know where to ask for help.

So based on your guidelines, I got this :

std::tie(mesh, rendering_params) = fitting::fit_shape_and_pose(morphable_model_with_expressions, landmarks, landmark_mapper, image.cols, image.rows, edge_topology, ibug_contour, model_contour, 1, eos::cpp17::nullopt, 30.0f);
const Eigen::Matrix<float, 3, 4> affine_from_ortho = fitting::get_3x4_affine_camera_matrix(rendering_params, image.cols, image.rows);
const core::Image4u isomap = render::extract_texture(mesh, affine_from_ortho, core::from_mat(image));

I got this (I think it is related to this issue #94 :

Since Mr. Morgan is a bit with his mouth open (but I guess not as much as the mesh). But I would like to have the mesh and texture complete, and not with a "hole" in his mesh/isomap.

Now the other two face images :

The fitting is much better (keept the same landmarks).

I know that I can "force them" to close their mouth in the mesh, but it looks very weird. So I would like to keep their expression, both mesh and isomap.

Thank you so much!

[patrikhuber]

Hi,

If you're fitting people in neutral expressions, then you could just fit without the expression fitting. I think eos doesn't have a function out-of-the-box that does "all" the fitting (shape, pose, contour) without expressions, but you can have a look at what fit_shape_and_pose(...) does and remove the part from it that estimates expressions.

Also, these mouths seem quite wide open, so again, I would recommend checking (visualising) your landmarks first.

But I would like to have the mesh and texture complete, and not with a "hole" in his mesh/isomap

I'm not sure what you mean by "hole" here, but if you mean the open mouth, then see my comment above - also you can have a look (and/or save) the texture map returned by render::extract_texture(...) - see also eos::core::write_textured_obj(...). If you have questions on texture mapping, then there is loads of resources to be found on the internet using your favourite search engine.

If you want to get really good results, then 3D model fitting is not an easy topic, I would recommend that you either bring a lot of time (and you'll need to read and experiment lots) or(/and) look for some sort of mentor/supervisor - for example at your Uni, workplace, in your community, social media circles, or /r/computervision, just for a few ideas.

Also, I've asked GitHub if this repo can take part in the beta of their new "Discussions" feature, but I haven't heard back yet - perhaps we'll have a forum of sorts here some time soon. But even then you might have more luck asking these more general questions somewhere where a larger number of users checks the posts.

[virtualenv]

Dear Patrick,

You know, I have seen some create creations in Github, but I haven't seen any kind of support like yours. So thank you, for this great library, as well as your kindness and patience with all of us!

Best regards always!

[patrikhuber]

Thank you for your kind words!

[sunjunlishi]

@patrikhuber dear author,I saw that you had an example where you could turn any expression into a neutral expression. Why didn't you see it. Now that I have extracted the expression change parameters, it works. Now I need the texture to be able to change according to the expression,

[patrikhuber]

Hi @sunjunlishi, it looks like your post is quite off-topic in this thread. Perhaps you want to use the new forums in the Discussions section and post a question for the community there? I am not sure what you're asking though, so make sure to describe your question in a clear way.