Texture extraction when using the BFM2009

[ghost]

Hi! Just wanted to register a problem I am having with the fitting algorithm.

I run into no problem when running the demo fitting script located at https://github.com/patrikhuber/eos/blob/master/python/demo.py

I can also successfully produce the "bfm2009.bin" file from the Basel Model downloaded at their website, and I can generate perfect vertex coloured .obj files of human faces from a eos.morphablemodel.MorphableModel loaded from bfm2009.bin.

However, when, in the demo.py script, I change line 10:

model = eos.morphablemodel.load_model("../share/sfm_shape_3448.bin")

to load the above mentioned bfm2009.bin instead of sfm_shape, i run into a segmentation fault. I believe this is related with the mathematical optimization process.

I am using Ubuntu 18.04 with gcc 6.4.0.

[patrikhuber]

I can generate perfect vertex coloured .obj files of human faces from a eos.morphablemodel.MorphableModel loaded from bfm2009.bin.

When, in the demo.py script, I change line 10: model = eos.morphablemodel.load_model("../share/sfm_shape_3448.bin") to load the above mentioned bfm2009.bin instead of sfm_shape, i run into a segmentation fault.

I am a bit confused, these two statements seem contradictory to me. The first one says you can load the bfm2009.bin, the second one says you can't?

[patrikhuber]

Well, that's your own fault then. Think about what you're doing. The BFM2009 doesn't come with texture coordinates (uv-coords), so you're trying to extract texture with a model that doesn't have texture coordinates. That's pretty much impossible. See also for example my comment here. This is for example documented in the BFM2009 converter script.

[ghost]

The model loads fine. The Segmentation Fault happens latter on in the demo.py script. I believed the problem was in the fitting algorithm, but it actually happens in line 30, when extracting texture. I am loading a png image, and I did make sure the width and height are matching:

image = cv2.imread('./img.png')
image_width = image.shape[1]
image_height = image.shape[0]

The exact same code runs fine with the sfm model, so I think this is related to the model dimensions of bfm.

[ghost]

Hi @patrikhuber . I didn't know about this difference. Thanks for explaining it. I understand that the Segmentation Fault is not a problem from eos now.

That being said, this question is related to a more fundamental problem I am having. I wanted to fit the full Basel model, WITH the texture parameters. But the "demo.py" script does not do that, instead, it tries to extract the texture from the visible portion of the face in the image. This leads to bad, non-simmetrical results. Basel model solves this offering texture parameters. But they have to be fitted.

Is it possible to use eos to do this? To fit the shape parameters AND the texture parameters of a model like Basel's?

Sorry if the question is dumb. I am beginning in computer vision and could not find a way to do this with the fit_shape_and_pose function. My impression is that it is not possible to fit the texture coordinates of the Basel model in eos.

[ghost]

I don't know if I was clear in the above question. When I load Basel model, I can do

sample = model.draw_sample([1.0, -0.6], [1.0, 1.0])

And I get a mesh WITH color, in the sample.colors attribute. But, when using the fitting function, there is no resulting color in the mesh. Is it possible to fit the PCA color coefficients?

[patrikhuber]

I think you are in general confusing "texture extraction" (or "texture remapping") and "PCA texture model"/"texture fitting", also more correctly called PCA albedo model or albedo fitting. These are two completely different things.

eos indeed does do shape fitting (and texture extraction), but it does not contain a direct algorithm for albedo model fitting. If you just want a linear solve, that would be relatively easy to set up though (you could see Aldrian & Smith 2013, link is in the Readme). Actually there's even a first-run example of nonlinear albedo fitting in eos, you could have a look at fit-model-ceres. Beware that you're getting into a difficult topic though ;-)

[ghost]

I did know about the difference between the two concepts already, but since I could not find a way to do texture fitting in eos, I was trying texture extraction as a temporary solution.

Anyway, thank you very much for the references! I'm starting to think I will need to learn C++ in order to do Computer Vision effectively.

[patrikhuber]

I see!

Well you can get a long way with Python I think :-) Good luck!