BFM09 texture coordinates (from stn/BFM_UV.mat) are flipped?

[lvZic]

I use texture_coordinates from stn/BFM_UV.mat and texture_triangle_indices from" bfm2009_raw['tl'] - 1". However, the obj result of fitting-model-simple is unexpected as following:

following is part of converting model code:

Construct and save the BFM2009 model in the eos format:

model = eos.morphablemodel.MorphableModel(shape_model, color_model, vertex_definitions=None, texture_coordinates= uv_coords_raw, texture_triangle_indices = triangle_list.tolist()) # uv-coordinates can be added here

[lvZic]

The fitting-model-simple doesnot change except input parameters, so there must be something wrong about the converting bfm model. The shape,color basis,mean, ev are from 01_MorphableModel.mat. I checked the uv_coords_raw, size is 53490 x 2, range [0 1], and it seems fine. And the triangle_list = bfm2009_raw['tl'] - 1, which is from your code. I am confused about the problem and wonder why. Because I found there are so many issues about bfm model, however none of them given a solution. So I wonder whether the bfm could reconstruct successfully or jsut sfm can do using eos?

[lvZic]

I used bfm_model_2017, and the reconstructed obj seems same with the one by model 09. So what i missed?

[lvZic]

However, the above texture_map is flipped? so I guess thre is something wrong with texture_coordinates ?

[lvZic]

The above result with flipped UV_texture's v coordinate, the obj result is following:

It seems a bit normal now, but I wonder why this problem about flipped v corrdinate never happen before or what i missed?

[patrikhuber]

Hi @lvZic,

So I wonder whether the bfm could reconstruct successfully or jsut sfm can do using eos?

eos works fine with the BFM. But since the BFM doesn't come with official texture coordinates (the older versions at least), I suspect that not a lot of people (including me) have used uv texturing with that model.

Yes, the flipping can happen - I think some people define UV coords with the origin at the top-left, others define the origin at the bottom-left. As you've figured out, you can just flip the y-component of the uv coords (i.e. new_y = 1 - old_y) to adjust this.

Also it looks like when you're displaying your result in MeshLab, you're actually overlaying the vertex colouring (probably the mean face in this case) with the texturing. If you just want to visualise the texturing in MeshLab, you have to fiddle around with the checkboxes a bit.

I think this can probably be closed?

[lvZic]

Hi @lvZic,

So I wonder whether the bfm could reconstruct successfully or jsut sfm can do using eos?

eos works fine with the BFM. But since the BFM doesn't come with official texture coordinates (the older versions at least), I suspect that not a lot of people (including me) have used uv texturing with that model.

Yes, the flipping can happen - I think some people define UV coords with the origin at the top-left, others define the origin at the bottom-left. As you've figured out, you can just flip the y-component of the uv coords (i.e. new_y = 1 - old_y) to adjust this.

Also it looks like when you're displaying your result in MeshLab, you're actually overlaying the vertex colouring (probably the mean face in this case) with the texturing. If you just want to visualise the texturing in MeshLab, you have to fiddle around with the checkboxes a bit.

I think this can probably be closed?

thanks for your reply, and i want to use eos to ectract identity and expression coef. the problem about flipped y-component of the uv coords is fixed. Now I used deep3d's uvmat(https://github.com/microsoft/Deep3DFaceReconstruction/blob/master/utils.py), but i am focusing on another problem, which i wonder why. the above texture_result is weird and both bfm09 and bfm17 model generated the same result. I have checked the basisi,mean,eigen coef of shape,color and expression coef, and found no mistake. Can u give me advice which i can missed?

[patrikhuber]

Do you have an example (maybe from their repository) of how Deep3D's UV mapping is supposed to look like?

I would double-check whether Deep3D's UV mapping uses a different triangulation for the texture triangles, in which case you need to set tti (texture triangle indices) correctly.

How does it look like when you visualise the resulting model in MeshLab with this texture?

[lvZic]

@patrikhuber the reason of above result is about cmake compile option, when the relwithdebinfo mode is set the result is weird as above, and debug or release mode works fine. And then i found the different units problem about bfm2009 model and fixed it by https://github.com/patrikhuber/eos/issues/255. However, the texture now is still abnormal as following:

the result in meshlab is :

[lvZic]

following is tri from DEEP3D(i used by tri = tri - 1 and swap second and third index):

following is UV mapping of DEEP3D (which i get from https://github.com/anilbas/3DMMasSTN/blob/master/util/BFM_UV.mat):

[lvZic]

I found the origin BFM09 model (53490 vertex) without expression can generate following result, however the color and shape position don't match correctly. Then i found a confusion about idBase and shapePC (DEEP3D combine shapePC*shapeEV as idBase, and eos use shapePC, shapeEV separately). So, the final is result is following:

[lvZic]

@patrikhuber Now, i generate coutours.json with corresponding vertex idx from bfm2009model, topology.json ,and run fit-model with bfm09_frontface_model(35709 vertex, without swaping indeces of triangles ). Final result is following:

Can u give me advise about what i missed?

At the same time, the result of fit-model-simple with the same bfm09_frontface_model generate more normal result as following:

I guess if there is something wrong with topology.json or contours.json?

[patrikhuber]

@lvZic Good to hear you're making progress on fixing your issues. If I understood your last post correctly, the fitting of the front-facing contour is not working well anymore? It looks like you created the topology and contour json files yourselves for your BFM model. I would start by double-checking / debugging these. You can start by visualising the front-facing contour correspondences to check if that's alright.

[lvZic]

@patrikhuber after re-recheck the contour json file, i recreate it using keypoint info from facemodel_info.mat in deep3d, then result is normal as following: The difference is the older contour json file is created by replaceing bfm2009_model_contours by corresponding vertex index bewteen front_face model and origin bfm09 model，and the new one is created by using keypoint info from facemodel_info.mat which contains 68 landmarks' vertex index. In my opinon, these two should be same....

Anyway, the latest result is normal except for mismatch between color and shape. Can u give me advice about what i missed? thanks a lot!

[patrikhuber]

model_contours.json should contain the vertex ids for the right and left vertices that correspond to the "2D" (ibug) landmarks. If you want to use the contour fitting, it may help you to read the relevant sub-chapters in my thesis that explain the contour fitting in eos? And it doesn't hurt to have a bit of a look at the code to get a broad understanding of how both the front- and back-facing contour fitting work, so you can define the contour vertices correctly. Also if you don't need it, you can disable the contour fitting too - for example fit-model-simple runs without the contour fitting.

As for the "mismatch" you're seeing, first, you're still overlaying vertex colouring and texturing in MeshLab, see my comment further above. Second, the "mismatch" comes from inaccurate fitting and it is expected to some degree. Remind yourself that you are only fitting to 68 (or slightly less) landmarks. fit-model does not do any dense fitting. If you want more accurate results, you can use more landmarks, or you can use eos's result to initialise a more dense fitting approach. Since you're using a different model, you can also play around with the shape regularisation a bit, you need to find a good value for your particular model where your fitting results can deviate from the mean enough, but not too much to generate unrealistic faces.

[lvZic]

model_contours.json should contain the vertex ids for the right and left vertices that correspond to the "2D" (ibug) landmarks. If you want to use the contour fitting, it may help you to read the relevant sub-chapters in my thesis that explain the contour fitting in eos? And it doesn't hurt to have a bit of a look at the code to get a broad understanding of how both the front- and back-facing contour fitting work, so you can define the contour vertices correctly. Also if you don't need it, you can disable the contour fitting too - for example fit-model-simple runs without the contour fitting.

As for the "mismatch" you're seeing, first, you're still overlaying vertex colouring and texturing in MeshLab, see my comment further above. Second, the "mismatch" comes from inaccurate fitting and it is expected to some degree. Remind yourself that you are only fitting to 68 (or slightly less) landmarks. fit-model does not do any dense fitting. If you want more accurate results, you can use more landmarks, or you can use eos's result to initialise a more dense fitting approach. Since you're using a different model, you can also play around with the shape regularisation a bit, you need to find a good value for your particular model where your fitting results can deviate from the mean enough, but not too much to generate unrealistic faces.

thanks for your help. and i will close this issue.