MrTornado24 / Next3D

[CVPR 2023 Highlight] Next3D: Generative Neural Texture Rasterization for 3D-Aware Head Avatars
https://mrtornado24.github.io/Next3D/
470 stars 29 forks source link

Question about the metrics (AED, APD, ID) #40

Open liguohao96 opened 4 months ago

liguohao96 commented 4 months ago
  1. Confusion about AED and APD. Evaluation of AED and APD are found both in https://github.com/MrTornado24/Next3D/blob/bdab84fc12cb61bd1002a3efb2c0a707a1139d97/evaluation.py#L92 and https://github.com/MrTornado24/Next3D/blob/bdab84fc12cb61bd1002a3efb2c0a707a1139d97/cal_3dmm_distance.py#L133. evaluation.py calculate AED/APD with math.sqrt(sklearn.metrics.mean_squared_error(np.array(exps), np.array(exps)[:, :50])) cal_3dmm_distance.py calculate AED/APD with AED = np.mean(np.abs(s_exp - t_exp)) these two are obviously different evidenced as below:
    >>> a = np.random.default_rng(0).random((2,4))
    >>> a
    array([[0.63696169, 0.26978671, 0.04097352, 0.01652764],
       [0.81327024, 0.91275558, 0.60663578, 0.72949656]])
    >>> b = np.random.default_rng(1).random((2,4))
    >>> b
    array([[0.51182162, 0.9504637 , 0.14415961, 0.94864945],
       [0.31183145, 0.42332645, 0.82770259, 0.40919914]])
    >>> math.sqrt(sklearn.metrics.mean_squared_error(a, b))
    0.5001152789168666
    >>> np.mean(np.abs(a - b))
    0.42166963796730605

    besides, the whole util module is missing.

  2. Evaluation of ID is totally missing! Please release a detailed code or guide of computing ID (especially FR net weights and preprocessing). I have followed description from your paper and utilize resnet-50 from arcface_torch and got 0.4386 ID score (far from 0.84 reported in paper) with publised next3d_ffhq_512.pkl. The render images seems OK. next3d_id