search

ShenhanQian / VHAP

A complete head tracking pipeline from videos to NeRF/3DGS-ready datasets.
Other
88 stars 12 forks source link

Real-Time Rendering from Raw Video #4

Closed MoniruzzamanMd closed 1 month ago

MoniruzzamanMd commented 1 month ago

Thank you very much for your excellent work!

I noticed that GaussianAvatars offers real-time rendering from processed data. However, I’ve found that when starting with raw video, step 2 (Align and track faces) can be quite time-consuming. Could you provide insights on how to achieve real-time rendering directly from raw video?

ShenhanQian commented 1 month ago

Hi, it is fast to render FLAME if the parameters are known. However, this repo uses photometric optimization to obtain the parameters. This process can be time-consuming because we run many iterations for the best quality.

If you want faster optimization speed, you can try to lower num_steps. But then you may have convergence problems.

Generally speaking, our pipeline does not target real-time optimization because we use Adam, a first-order optimizer. And PyTorch does not provide a complete support of second-order optimizers.

MoniruzzamanMd commented 1 month ago

Thank you for the clarification!