search

barisgecer / GANFit

Project Page of 'GANFIT: Generative Adversarial Network Fitting for High Fidelity 3D Face Reconstruction' [CVPR2019]
http://openaccess.thecvf.com/content_CVPR_2019/html/Gecer_GANFIT_Generative_Adversarial_Network_Fitting_for_High_Fidelity_3D_Face_CVPR_2019_paper.html
GNU General Public License v3.0
627 stars 65 forks source link

Input of the Texture GAN #6

Closed RegisWu closed 4 years ago

RegisWu commented 4 years ago

Hi @barisgecer, thanks for sharing this interesting work! The ideas are novel and the results are very impressive!

In the paper, you train a Texture GAN for with texture parameters (pt) as input generating UV texture maps for each ID with the ground-truth provided in the texture dataset.

After reading the paper, I wonder how to obtain the mentioned texture parameters pt? Is it estimated by 3DMM models? And is it the color value of each vertex or something else?

As a beginner in 3D face topic, I would appreciate it if you could provide more details about it, which would be very helpful to me!

I am looking forward to your reply!

Best regards.

barisgecer commented 4 years ago

Training Texture GAN is a separate process than parameter fitting. First, we train the GAN with random Gaussian distribution as input. Then use this GAN as our texture model and do the fitting.

The texture model is on 512x512 UV images. But we sample from that UV for each vertex during optimization for simplicity of the differentiable renderer. Nevertheless, we have predicted parameters to generate UV at full resolution after the fitting.

Hope this helps. Baris

RegisWu commented 4 years ago

Thanks for your quick reply!

I understand the training pipeline and the application of the Texture GAN.

However, I am still confused about 'how to obtain the texture parameters pt for a specific sample'. The pt is used as the input to the Texture GAN, which should encode identity information to guide GAN to generate personal specific texture.

Is pt provided in the dataset or extracted by other tools? Or discover by the backward process?

Best regards.

barisgecer commented 4 years ago

Short Answer: yes by 'backward process'

Long Answer: p_t is optimized by GANFit optimization. It is basically analysis-by-synthesis (optimization-based) face reconstruction. This means, iteratively, we synthesize a texture (along with the shape and other parameters), render it, compute respective loss functions and backpropagate the error to the parameters (including p_t). So that it is updated towards the more similar texture to the input image. Basically, these parameters start randomly and converge toward the ideal face reconstruction in terms of loss functions. If you can look at the paper, you will have a deeper understanding.

RegisWu commented 4 years ago

Many thanks for your reply! It really helps address my confusion!