search

SimonGiebenhain / NPHM

[CVPR'23] Learning Neural Parametric Head Models
https://simongiebenhain.github.io/NPHM/
Other
212 stars 17 forks source link

Training on Facescape Dataset #7

Closed aejion closed 1 year ago

aejion commented 1 year ago

Hi, it's a fantastic work!

I have a question about training NPHM on Facescape Dataset. First, I have prepare the neutral expression data to train the Identity Network, and it can obtain a nice result as the following image shown.

image

facescape_neutral.zip

And then I use the registered mesh of Facescape dataset and run the scripts/data_processing/sample_deformation_field.py to prepare samples, then train the forward deformation fields. But the results on the training set seems to be wrong. Could you please help me figure out the problem?

image image

facescape_expression.zip

Thanks!

SimonGiebenhain commented 1 year ago

Hi,

I assume that by "seem wrong" you mean that there are tiny ripples/imperfections in the expression reconstruction? Or is it the fact that the expression is not captured accurately?

Regarding the first problem, I guess you generated the result while the .training attribute of the deformation network set to True. Here you can see that during training I add a bit of noise to the "compressed" identity code. Please note that I don't know if that is a good or bad thing, but there was little time and I tried to avoid overfitting.

Regarding the second problem: Maybe the network just needs to train longer. In general learning forward deformations in the style of NPHM requires detailed and consistent (!) registrations. I believe that learning backward deformations, like i3DMM and especially ImFace, is promising and does not require registrations at all.

aejion commented 1 year ago

Thanks for your reply!

During my training process, I encountered both of the issues you mentioned. I suspect that the problem might be due to noise added to the "compressed" identity code during training. And I use the consistently registered mesh to train the model, I will try to train longer. Thanks for your help!