Run-time of the fitting

[bigdimboom]

What's the performance/frame rate of current fitting algorithm?

[patrikhuber]

You can check Section 4.7 of my PhD Thesis for some concrete numbers. In short, it's around 60 to 1000 frames per second, depending on your hardware, the model, and parameters.

[bigdimboom]

Cool. What other papers do you recommend if I want to understand your code thoroughly?

[patrikhuber]

I'd say my thesis and probably the original Blanz & Vetter paper about 3DMMs. Then there's dozens of other great papers, also recent ones, if you want to get into more details. Just check the major conferences and journals (PAMI, SIGGRAPH, CVPR) and you'll find the major 3DMM fitting works.

[bigdimboom]

I feed my web camera image and dlib landmark detection to the fit_model example. The FPS is about 10 fps.(Raw web camera is 15 fps.) I also tried feeding data to simple_model_fit, the performance is good but some pixels in the background fit on to my face . What are the ways to improving performance with the model_contour enabled?

This is the fitting function I used.

[patrikhuber]

It doesn't look like eos's fit_shape_and_pose is the issue. Particularly if you only get 15 fps with raw webcam. I'd say run a more thorough profile run in RelWithDebug mode.

[bigdimboom]

interesting..... I changed my web camera to 30fps. The fitting program are now running at 15-18 fps. But it dropped to 7-10 fps when I move my mouth dramatically. I went through your code quickly yesterday. It looks like a lot of them are GPU-able, for example, the function "extract_iosmap".

frame rate also drops when I move my head, but the mesh fits my face with good angle. Great work!

[patrikhuber]

extract_texture is not included in my measurements - it's not a fitting function, it's more of a helper function that's normally not needed. This will take most of your run time actually. Yes, this could be done with OpenGL in principle :-) But we want to keep the core of eos free of a dependency on OpenGL (and framebuffer).

Thanks!

[bigdimboom]

ok, I moved all rendering related code to opengl. Now I get 15 - 20 fps when the settings of iterations: 3, lamda: 5.0. Could it be faster if i switch to Ceres solver?

[patrikhuber]

I'm pretty sure Ceres wouldn't really be faster as we're just solving linear systems (and doing the contour correspondence fitting, which you can't really easily do in Ceres). I suggest you run the profiler and see where your time-sink is, you should get more fps when just doing the fitting. Something else must be slow or maybe you're not running with optimised compiler flags. PS: lambda=5 is quite small, I'd recommend something like 60.

[bigdimboom]

Sorry, it was a typo. 50 the default value is the one I used. OK. I will re-profile my code again.

[bigdimboom]

Very interesting.... After I changed the lamda to 60 in stead of the default 50. The frame rate is now 30 now for a HD camera

[patrikhuber]

@bigdimboom To be honest I think you must've changed something else, unknowingly - or maybe your build settings changed or something like that and you didn't notice. The regularisation can influence the run time slightly, in principle, but it wouldn't be a difference from 15-20 to 30 fps.