Issue when using the BFM

[ngasp]

After using the conversion scripts available (convert_bfm2009_to_raw_binary.m ; bfm-binary-to-cereal ), i tried to run your fitting algorithm with the BFM converted file (bfm2009.bin) but received a seg fault. (./fit-model -m bfm2009.bin -p ../share/ibug2did.txt -i data/image_0010.png -l data/image_0010.pts) What could be going wrong?

[patrikhuber]

-p ../share/ibug2did.txt

These are mappings from the 68 ibug landmarks to vertex IDs in the Surrey Face Model. Did you adjust them to the correct vertices of the BFM? (The BFM comes with some kind of feature mapping files too, shouldn't be too hard to adjust that.)

[ngasp]

Sorry for the late response. Even with the correct vertex mapping it fails, i think it is because the conversion script given a warning concerning the number of texture vertices, the seg fault happens when : // Extract the texture from the image using given mesh and camera parameters: Mat isomap = render::extract_texture(mesh, affine_from_ortho, image); is called. Sorry to bother with this.

[patrikhuber]

Aah. I'm not sure about the warning (what does it say?) - but I don't think the BFM has texture coordinates (uv-coords) at all, does it? Without them, one can logically not do texture extraction or remapping.

Would you mind sharing your mappings? I'd have a look and see if I can improve the error-messaging. (Are you using Debug or Release builds?)

If you need texturing, then you will have to create texture coordinates for the BFM (or possibly ask the creators for it, I'm sure they have them!). I think Blender can output texture coordinates for meshes, otherwise maybe Matlab.

[patrikhuber]

Can this be closed?

[ngasp]

Sorry, the warning is : Loading shape PCA basis matrix with 160470 rows and 199 cols. Loading colour PCA basis matrix with 160470 rows and 199 cols. Warning: PCA shape model's data dimension is different from the number of texture coordinates given. The converted model is still saved, but most likely not work correctly for texturing.

Yeah, i can share the mappings :

          ; 1 to 8 are the right contour landmarks
 9  48188 ; chin bottom
          ; 10 to 17 are the left contour landmarks
18  39227 ; right eyebrow outer-corner (18)
19  39773 ; right eyebrow between middle and outer corner
20  40090 ; right eyebrow middle, vertical middle (20)
21  40266 ; right eyebrow between middle and inner corner
22  40434 ; right eyebrow inner-corner (19)
23  41173 ; left eyebrow inner-corner (23)
24  41338 ; left eyebrow between inner corner and middle
25  41577 ; left eyebrow middle (24)
26  41872 ; left eyebrow between middle and outer corner
27  42272 ; left eyebrow outer-corner (22)
28   8282 ; bridge of the nose (parallel to upper eye lids)
29   8303 ; middle of the nose, a bit below the lower eye lids
30   8314 ; above nose-tip (1cm or so)
31   8321 ; nose-tip (3)
32   5491 ; right nostril, below nose, nose-lip junction
33   6913 ; nose-lip junction
34   8330 ; nose-lip junction (28)
35   9751 ; nose-lip junction
36  11039 ; left nostril, below nose, nose-lip junction
37   1957 ; right eye outer-corner (1)
38   4017 ; right eye pupil top right (from subject's perspective)
39   5309 ; right eye pupil top left
40   6086 ; right eye inner-corner (5)
41   4934 ; right eye pupil bottom left
42   3774 ; right eye pupil bottom right
43  10344 ; left eye inner-corner (8)
44  11244 ; left eye pupil top right
45  12531 ; left eye pupil top left
46  14084 ; left eye outer-corner (2)
47  12931 ; left eye pupil bottom left
48  11899 ; left eye pupil bottom right
49   7201 ; right mouth corner (12)
50   6284 ; upper lip right top outer
51   7313 ; upper lip middle top right
52   8345 ; upper lip middle top (14)
53   8874 ; upper lip middle top left
54  10411 ; upper lip left top outer
55  10811 ; left mouth corner (13)
56   9266 ; lower lip left bottom outer
57   9020 ; lower lip middle bottom left
58   8375 ; lower lip middle bottom (17)
59   7730 ; lower lip middle bottom right
60   7460 ; lower lip right bottom outer
          ; 61 not defined - would be right inner corner of the mouth
62   7063 ; upper lip right bottom outer
63   8223 ; upper lip middle bottom
64   9643 ; upper lip left bottom outer
          ; 65 not defined - would be left inner corner of the mouth
66   9271 ; lower lip left top outer
67   8239 ; lower lip middle top
68   7465 ; lower lip right top outer

1 -8 : 27302 46587 45866 2445 1260 721 316 27431

10 -17:
27689 16312 15943 15450 14313 50480 49788 27818

And for texturing, i don't really need it now, i just wanted to see if there is any difference when using the BFM instead of the surrey model.

Sorry for taking so much time to answer, and thank you for the willingness to help :)

[patrikhuber]

Thank you very much for sharing the mappings! I'll try to play around with the BFM in the next days/weeks too!

[ngasp]

Turns out if i use the fit-model-simple and not the one i was using, fit-model, the bfm works, but the results are not exactly good, maybe bad annotation? i'll try and play around with the code in the next days, if i get any progress i'll update here

[patrikhuber]

Hm, maybe it's an issue with the contour landmarks or their mappings/definitions then! It can also be that the contour landmark fitting algorithm behaves differently with the BFM. I'm currently trying to get something else to work with involves the BFM so in the process of that task, I might also be able to investigate fitting with the BFM in general a bit closer.

See also #102 which might be related.

[patrikhuber]

I just wanted to inform here that it looks like above posted mappings have some major issues. I suggest anyone wanting to use them to double check them. I hope that we can have a corrected one some day and include it into the repo.

[patrikhuber]

I've added some mappings for ibug landmarks in 5dbff1ec4a134f57511413361a5c7104f2b9f0d6, that should be a good starting point.

[dbersan]

I had the same issue when trying to use the Basel Face Model, as described here.

The problem is that BFM does not have any texture coords mapping, so at this line it breaks, because mesh.textcoords is empty...

I was wondering if it would be trivial to fix this. Like, if there are no texture coordinates, they are automatically calulated, like maybe using isomap projection? Do you have a suggestion?

[patrikhuber]

You definitely cannot extract the texture without having texture coordinates, that's not an issue of the eos code.

Yes you can definitely compute texture coordinates for the BFM using isomap projection or something like that. It's what I would do.

[ElliotSalisbury]

I'm not a hundred percent sure, but the ibug_to_bfm.txt in https://github.com/patrikhuber/eos/commit/5dbff1ec4a134f57511413361a5c7104f2b9f0d6 might have the left and rights flipped.

I was getting pose fittings with the face pointing away from the camera, and by swapping the left and right indexs, the face now points towards the camera. Maybe this is related to the y coordinates being mirrored mentioned in this thread: https://github.com/patrikhuber/eos/issues/102

[patrikhuber]

@ElliotSalisbury Thanks for noting this! I think when I did these mappings I checked quite carefully, so I'd say I'm 95% certain they're correct. But it's certainly possible that I made a mistake!

However I think you are maybe the "victim" of something else: The BFM has its triangle orientation "the wrong way round" - i.e. it specifies front-facing vertices in CCW order, whereas the de-facto standard in graphics (e.g. OpenGL) seems to be CW (or was it the other way round? In any case, the BFM order is non-standard). So for example if you do a simple experiment and display the mean BFM shape as obj in Meshlab, it looks like it's the wrong way round - when in fact it is not - it's a kind of optical illusion, it looks the wrong way round because the lighting is all weird, which is because the normals are the wrong way round (because the triangles are the wrong way round).

I'm not sure why the Basel group chose to specify the triangles like this :-) Maybe it would make sense to flip the triangle orientation in the bfm-to-eos conversion scripts? (That is, if I'm really correct with all this!). I'm happy for any input you might have.

[ElliotSalisbury]

Yeah, that's totally the effect I was seeing, I've just run some tests to confirm it, my bad!

I think it makes sense to flip the triangle orientation in the conversion script.

As a side note: I've only just started playing around the bfm, and am I correct in thinking there are no blendshapes defined for the bfm? as I can't call fit_shape or fit_shape_and_pose without them. Perhaps the fitting code could check whether we have blendshapes e.g.:

in fitting.hpp, the fit_shape_and_pose_multi function:

    ...

    VectorXf current_pca_shape = morphable_model.get_shape_model().draw_sample(pca_shape_coefficients);
    vector<VectorXf> current_combined_shapes;
    vector<eos::core::Mesh> current_meshs;
    for (int j = 0; j < num_images; ++j) {
        VectorXf current_combined_shape = current_pca_shape
        //if blend shapes exist, use them, otherwise just use pca shape
        if (blendshapes.size() > 0) {
            current_combined_shape += blendshapes_as_basis * Eigen::Map<const Eigen::VectorXf>(blendshape_coefficients[j].data(), blendshape_coefficients[j].size());
        }

        ...   

There's also no model_contours or edge_topology either, were these generated for the sfm manually? or is there a script?

[patrikhuber]

Yeah, that's totally the effect I was seeing, I've just run some tests to confirm it, my bad!

Hehe! Cool! Okay, I might add that flipping in the conversion then. Actually I played around with the BFM again too 1-2 month ago but got stuck because it doesn't have texture coordinates either, and generating them wasn't so trivial. I won't have time to continue this in the next 4 weeks as I'll be submitting my PhD, but after that, it's definitely high up in my list to help get the BFM to work properly with all functionality of eos.

there are no blendshapes defined for the bfm

Correct, there aren't - the BFM is just an identity model. There are some groups that learn an expression basis (or blendshapes) from FaceWarehouse and add that to the BFM to model expressions. Also your suggestion of the if-check in the fit_shape_and_pose_multi sounds great to me as a quick solution! (or even a long-term solution ;-) )

model_contours

That should be fairly easy to create as you just need to mark a number of vertices along the left and right contour, which correspond to ibug landmarks.

edge_topology

This one you can actually take from https://github.com/waps101/3DMM_edges. I used their script to generate the file for the SFM actually.

I'll have a quick look if I have these laying around for the BFM somewhere.

[patrikhuber]

Here's a model_contours file for the BFM - no guarantees though, did that in February and don't exactly remember how well I did it (and if I even tested it!). (remove the .txt at the end obviously, GitHub doesn't allow to upload .json files). model_contours_bfm.json.txt

You can download bfm_edge_topology.json here: https://patrikhuber.ch/files/github/bfm_edge_topology.json. If this doesn't work for some reason then I can also send you the intermediary files and/or the script, but I think it should be just fine.

Btw if these files work I would be very happy to put them in the eos wiki. (I think not in the repository as I'd like to keep the share/ directory fairly lightweight).

[oopming]

hi patrikhuber: I'm using BFM instead of SFM. I run bfm-binary-to-cereal.cpp, but Where can I get texture-coordinates obj_file? I'm looking forward to your help,thank you!

[patrikhuber]

Hi @oopming, if you read above posts in this issue, you should find all the information that you need about what's up with texture coordinates and the BFM. Thanks.

[patrikhuber]

I just committed a new and tested version of a script that converts the BFM2009 to the eos format: 7d918d911c6b9da7d2a06b9d4b441a86bcce8d25 I think the old script didn't convert the eigenvalues correctly. The new script is now tested and produces the same samples as in the BFM2009 AVSS publication from the group in Basel.

I have also updated the eos wiki and will close this. Anyone, feel free to open a new issue if there are still problems with the BFM2009. Thanks!