search

patrikhuber / eos

A lightweight 3D Morphable Face Model library in modern C++
Apache License 2.0
1.92k stars 598 forks source link

Bad result with SFM, should I mirror y coordinates? #102

Closed wguo68 closed 7 years ago

wguo68 commented 7 years ago

I used BFM and SFM morphable model to do shape fitting without blendshape model. All without edge constrains. The fitting results (with different weight) is too bad for eos although I tried with different shape prior weights. I might made some mistakes ,but I haven't found the bug.

eos

eos2

I think there might be two reasons: 1) For BFM model ,the y-coordinates of landmarks are mirrored as y = img_height-y+1 ,that is small y for bottom pixels . For SFM model, y is not mirrored as eos did. 2) I also noticed the z-coordinates of 3d point in the model is positive in BFM and negative in eos model.

BTW ,both used the normalized shape basis.

BFM mean model : v -57239.015625 42965.6875 80410.125 v -57191.5859375 42698.578125 80437.9140625 v -57140.375 42431.98828125 80463.890625 v -57090.83203125 42095.953125 80453.8359375 v -57034.40234375 41760.921875 80440.7421875 v -56967 41365.0859375 80406.1171875 SFM mean model: v -52.30980835213897 -52.65997675631559 -70.24066997426419 v -49.95127530552107 -58.01823217565432 -74.77978278239506 v -43.17182259840274 -64.8964190446127 -64.12540017016369 v -17.51681156632089 -80.23333476491459 -36.13685001099138

What should I do with this problem? Another question: which MM model is better (more data) for SFM and BFM?

patrikhuber commented 7 years ago

Thanks for the details! I think the fact that y and z are flipped shouldn't matter, because it will just be flipped in all estimated pose matrices as well (rendering-parameters).

I think this might be an issue of the range of the values in the basis, and regularisation. Can you try to set the regularisation much higher? E.g. 1000x higher?

I'm planning to test the BFM soon as well (next 1-3 weeks), hopefully then I'll discover this as well and have more time to look at it.

wguo68 commented 7 years ago

These are results with different weights for SFM model. eos2

patrikhuber commented 7 years ago

I meant trying different regularisation weights for the BFM. Did it fix your issues?

Are you using the correspondences from ibug2did.txt for the BFM or SFM? If you're using them for the SFM, then your drawing code is incorrect, as the correspondences are correct. If you're using them with the BFM, then you of course need to adjust the mappings as the BFM has completely different vertex ids.

wguo68 commented 7 years ago

I tried with different weights for BFM model. The result is : bfn

I am sorry I made a mistake in drawing code as you said when showing feature vertices of SFM model. eos4

Then I used vertices with its index larger than 845. and the results are :
eos4

So in the experiment,BFM model is better than SFM model. We should use larger facial MM model with much more face models such as 1000-5000 people with different expressions.

wguo68 commented 7 years ago

I used 68 feature points to do fitting with BFM model and sfm model ,the result is : bfm

It seems both are not like the real face!

patrikhuber commented 7 years ago

It seems to me like it's working fine. Can you draw the original 68 landmarks onto the image and then the fitted model, for example with draw_wireframe(...)?

hwdong commented 7 years ago

Here are their wireframe picture, I don't think they are okay. bfm

eos2

patrikhuber commented 7 years ago

In the second example, you definitely have to increase the regularisation.

As for the first example, I think you may need to improve your landmark-vertex mappings?

Also, the input renderings, are they created using a perspective camera? As eos uses a scaled orthographic camera model for most of its algorithms, I recommend the input data is also rendered/generated under orthographic projection.

hwdong commented 7 years ago

I used the scaled orthographic camera . What do you mean by input data (input rendering)? For the second example, the result can be seen above by increasing the regularisation. For the first example, here are the cooresponding vertices: bfm_vid

Another test: eos

patrikhuber commented 7 years ago

What do you mean by input data (input rendering)?

I mean the data you're running the fitting on. If you're running it on the example images/scans included in the BFM zip file, they may (or may not) be created with a perspective camera? I'm not sure about that, but if they are, eos's scaled orthographic camera model will not be able to estimate it too well, if the perspective effect is strong.

You have to keep in mind what you're doing is fitting to landmarks. The shape will naturally not be so precise, as you only fit to sparse landmarks. So the corresponding 3D vertices should fit quite precisely to the landmarks, and everything else is "approximated" from the model.

hwdong commented 7 years ago

I see . It is not good enough with landmarks only.

patrikhuber commented 7 years ago

It may well be! It always depends on the use case :-)

patrikhuber commented 7 years ago

There were probably some mistakes in the old BFM2009 to eos converter scripts. I just commited a new and tested version: 7d918d911c6b9da7d2a06b9d4b441a86bcce8d25

I'll close this, as I don't think there is a specific issue with the eos library left here. Otherwise, feel free to re-open.