search

patrikhuber / 4dface

Real-time 3D face tracking and reconstruction from 2D video
https://www.4dface.io
Apache License 2.0
793 stars 248 forks source link

Model fitting to videos #23

Closed codr1 closed 7 years ago

codr1 commented 7 years ago

Quick question on how things work - it seems that reset the landmarks every frame. So does the fitting improve with multiple frames? Does the PCA get refined? Or is it each frame for itself?

patrikhuber commented 7 years ago

4dface is a very simple demo from around a year ago that performs per-frame fitting, as you correctly described. The fitting function can take initial parameters (for example from the previous frame), in which case it converges faster, but it's still per-frame (and I don't remember if I finished implementing this - but it should be easy to find out). We also have a multi-frame fitting that fits to multiple frames at once, and thus improves the shape estimates. This is currently in a separate branch here, and we've got very promising results with it. It will eventually be merged to the eos master branch, but it'll take a while more.

Thanh-Binh commented 7 years ago

@patrikhuber are you sure that multiframe fitting in this branch works because ig was updated a month ago?

patrikhuber commented 7 years ago

@Thanh-Binh I'm not sure I understand your question. That is the multi-frame fitting that we have been using and it works very well for us.

Thanh-Binh commented 7 years ago

@patrikhuber how many frames do you need for each fitting call?how about accuracy vs runtime?

patrikhuber commented 7 years ago

@Thanh-Binh We determined 10-20 to be a good number. The accuracy increases quite steeply when going from 1..2..3..5..10 frames. From then on, it flattens a bit. I can't give concrete runtime numbers from the top of my head but our implementation is pretty fast and IIRC it's something like linear in the number of images.

Thanh-Binh commented 7 years ago

Very nice to hear! I will test it soon!

patrikhuber commented 7 years ago

I'm going to close this as it's not an issue per se, and multi-frame fitting is being worked on in above linked branch in eos.