search

gafniguy / 4D-Facial-Avatars

Dynamic Neural Radiance Fields for Monocular 4D Facial Avater Reconstruction
679 stars 67 forks source link

Huge Artifacts occured in cross-id face reenactment results #57

Open szh-bash opened 1 year ago

szh-bash commented 1 year ago

Model: 400k iters trained with 8x1080TI on Person_1 train dataset Test: Person_2 test dataset Simply modified basedir in load_flame_data(basedir,...) (eval_transformed_rays.py) from cfg.dataset to cfg.dataset.basedir[:-1]+'2' (which means "..../dataset/person_2/") and got cross-id face reenactment results below. We can easily find there're huge Artifacts in the generated video. Did i do anything wrong here? Or is the model behaving as expected? @gafniguy

https://user-images.githubusercontent.com/27895812/233761082-ddf517b2-6eae-403c-9a80-7f082351f142.mp4

Compared with Same-ID-Reenactment, it is found that the size of the Source face is entangled with the expression and posture parameters, and it tends to be adjusted to the Driver Face

https://user-images.githubusercontent.com/27895812/233763377-8443d762-d69a-41e1-bbfb-6ed2751585d2.mp4

szh-bash commented 1 year ago

But it does roughly transfered expression and pose from driver frame

alchemician commented 1 year ago

@szh-bash - did you figure it out?

szh-bash commented 1 year ago

@szh-bash - did you figure it out?

no further progress

gafniguy commented 1 year ago

Tips for cross subject reenactment would be to select a neutral expression looking frame from each, and then only transfer the deltas from that. This reduces the domain gap.

if the heads are very different in size between different models/identities, it could happen that with poses from one you will be rendering previously unseen regions, or get repeating artefacts in some regions due to positional encoding. Try to use just the rotation from the driving frame, but the intrinsics and translation from the actor.