search

tobias-kirschstein / nersemble

[Siggraph '23] NeRSemble: Neural Radiance Field Reconstruction of Human Heads
https://tobias-kirschstein.github.io/nersemble/
175 stars 7 forks source link

Expressions Transfer #3

Closed oijoijcoiejoijce closed 11 months ago

oijoijcoiejoijce commented 11 months ago

Great job with this - how would you suggest doing expression transfer between 2 different rendered human heads?

tobias-kirschstein commented 11 months ago

Hi,

Thank you for your interest in the NeRSemble project. With the current setup expression transfer won't work because both the time embeddings for the deformation field $\omega_t$, as well as the blend weights $\beta_t$ will learn totally different manifolds for different persons. Probably, the most straight-forward way to extend NeRSemble to allow expression transfer would be to use a FLAME tracking for the two subjects and adapt NeRSemble's architecture such that both $\omega_t$ and $\beta_t$ are decoded by an MLP from the respective FLAME expression parameters. That way, we learn to model the facial dynamics only based on the FLAME expression parameters which means that swapping in the expressions of a different person should give realistic renderings. This is a similar philosophy that other approaches such as AvatarMAV pursue. I would expect that this would equip NeRSemble with reasonable expression transfer capabilities, but also impact the view synthesis quality, as the MLP won't be as expressive as separately learned manifolds per subject.

Best, Tobias