DECA evaluation with REALY

[yjhong89]

Hi. Thanks for sharing great works! I am YJHong and currently developing my own 3d face reconstruction based on DECA.

After I've read evaluation protocol written in README, I have following questions for evaluating my model to REALY.

• When doing keypoint preparation,
  • DECA uses FLAME, but FLAME.obj uploaded in data/FLAME.obj is quite different from FLAME used in DECA. So should I place the DECA template file instead of using uploaded FLAME obj in this repo ?
  • Uploaded FLAME.obj differs in shape and global position while FLAME template used in DECA is positioned around origin (0,0,0)
  • Since barycenter txt file contains face index / bary center, it seems that can be utilized whatever FLAME template is loaded.
• When doing evaluation,
  • What the scales in the metrical_scale file means ? Are they generally used across 3dmm models ?
  • Currently, template_mask argument is set to None, but FLAME has vertex mask representing face region. Haven't you used this mask when evaluating DECA ?

Thank you. YJHong.

[czh-98]

1. For keypoint preparation.

   • I have updated the templates with FLAME_detail.obj, which can be used for evaluating detail models in FLAME template like DECA. The default pose for the template does not influence the results, the template is only used for extracting corresponding keypoints to the prediction. Hence, we only need to check whether the template shares the same topology as the prediction.
   • This repo uses a template because some methods might predict different triangles: e.g., 1/2/3 v.s. 2/3/1. In these cases, although it shows the same results, the barycenter will be wrong and thus influence the evaluation results.

2. For evaluation

   • The metrical scale is used to rescale the error into the original size of the scans. In this way, the shape difference is measured in proper metric units and reflects real-world differences.
   • We use the full-head topology for evaluation. The template_mask was used to evaluate the "prediction-to-scan" error (see Tab. 4 in our paper). Since our evaluation pipeline computes the scan-to-prediction error, a full-head topology can reach better metric results.

[yjhong89]

Thanks for kind answers! I have another question.

When evaluating DECA from REALY benchmark images, how did you crop evaluation images before feed into DECA encoder ?

• Just resize images from 512 to 224 or crop face from face detector and warp to 224 ?

[czh-98]

You can either resize the pre-processed images into 224x224 or set --iscrop as True (see https://github.com/yfeng95/DECA/blob/2d8864b34120d5a2c608dbce9aa5eec76e90cef9/demos/demo_reconstruct.py#L101). Since I notice some results from DECA tend to be wrong when I set iscrop as True, I set it as False for reconstruction and evaluation.