search

yfeng95 / PRNet

Joint 3D Face Reconstruction and Dense Alignment with Position Map Regression Network (ECCV 2018)
http://openaccess.thecvf.com/content_ECCV_2018/papers/Yao_Feng_Joint_3D_Face_ECCV_2018_paper.pdf
MIT License
4.96k stars 944 forks source link

Frontalization is a Non-Rigid Transform (Causes issues) #82

Open gsssrao opened 6 years ago

gsssrao commented 6 years ago

Just wanted to mention that the frontalization routine which has been implemented can be erroneous.

The way you currently seem to be doing it is as follow: 1) Compute the least square transform (solves for 3x4 matrix) between the output vertices and the original front facing face model (probably a Basel Face Model). 2) But, since the transformation you are computing is a general 3x4, it would also have shear, scale ... 3) Hence, the frontalized face would look different from the face before undergoing transformation. This is because the original front facing face model i.e canonical_vertices don't have the same face shape as the face which you want to frontalize.

"I completely agree that the current implementation frontalizes the mesh", but ideally you would want to frontalize it without changing its structure i.e by computing a rigid transform like a Procrustes. In the current implementation for example, say if you give it an oval face, it squashes it to a circle during frontalization.

developer-mayuan commented 6 years ago

@gsssrao I agree with you.

cleardusk commented 5 years ago

Dense points regression do not contain pose information, computing the least square transform is one solution of trade-off. This project 3DDFA also outputs 3x4 matrix, but without computing least square transform.

gsssrao commented 5 years ago

@cleardusk I agree that the network just regresses points and to compute the the fontalisation you will have to compute a transformation using these points (least squares of course is one such way).

But, what I have been pointing out is the fact that just computing 3x4 matrix using least squares would give rise to a non-rigid transformation. You can compute a rigid transformation if you change the code to compute say a Procrustes transform instead of a non-rigid transform.