search

Rubikplayer / flame-fitting

Example code for the FLAME 3D head model. The code demonstrates how to sample 3D heads from the model, fit the model to 3D keypoints and 3D scans.
http://flame.is.tue.mpg.de/
714 stars 108 forks source link

The performance problem of fitting FLAME model to a scan with nearly 1 million vertices #54

Open icewired-yy opened 4 months ago

icewired-yy commented 4 months ago

Hello. First of all, thank you for your impressive work and beautiful code.

I'm using the fit_scan.py to fit the FLAME model to my own scan. Unlike the tested scan in the repository, which has about 40K vertices, the scan I used has almost 1M vertices (more precisely, 1,334,321 vertices). And then I find that the fitting process will cost more than 3 hours (11099s).

I'm wondering if this situation is caused by my mistakes, or the performance of this fitting process do become worse when the number of vertices of target scan is increasing. Is it possible to accelerate this process when facing the scan with large amount of vertices?

TimoBolkart commented 4 months ago

You are right, first, the Chumpy code is not super efficient, and second, 1M vertices is huge. I suggest you either downsample the mesh to a lower resolution, or to run optimization with different smaller subset of scan vertices in each iteration (similar to batch training of networks). As the FLAME model has a much lower poly count, fitting the 1M vertices for the entire optimization has likely no benefit.

icewired-yy commented 4 months ago

Thank you very much for taking the time to respond to my inquiry. Your insights have been truly inspiring and helpful.

I have now switched to scans with fewer vertices, and the fitting results are quite satisfactory.

Your suggestion inspired me. Given that the number of vertices in the flame model is far less than 1 million, perhaps I could uniformly sample points in some way to construct several sub-models. Then, during each iteration of fitting, I could randomly select and fit these sub-models instead of directly using all the vertices for fitting.

Even if this approach is slightly more complicated than directly using a downsampled model, I would try implementing it in my spare time.

Once again, I greatly appreciate you taking the time out of your busy schedule to respond to my question. Your guidance has been invaluable:)