high reconstruct error on now challenge

[HITKJ]

Thanks for the work, it is very helpful. But I got some problem on test for now challenge. I tested the pretrained model that you supply on Now challenge and got much higher error on validation part of Now challenge.

(deep3d_pytroch_official_nexp - median: 1.286424, mean: 1.864963, std: 2.361429)

• I use MTCNN to detect 5 landmarks for every face in Now challenge.

• Align the face to the template mesh as you did in test.py

def read_data(im_path, lm_path, lm3d_std, to_tensor=True):
    # to RGB 
    im = Image.open(im_path).convert('RGB')
    W,H = im.size
    lm = np.loadtxt(lm_path).astype(np.float32)
    lm = lm.reshape([-1, 2])
    lm[:, -1] = H - 1 - lm[:, -1]
    _, im, lm, _ = align_img(im, lm, lm3d_std)
    if to_tensor:
        im = torch.tensor(np.array(im)/255., dtype=torch.float32).permute(2, 0, 1).unsqueeze(0)
        lm = torch.tensor(lm).unsqueeze(0)
    return im, lm

• Also，I set exp and angle as 0 to produce netural face mesh for every picture.

  coef_dict = self.split_coeff(coeffs)
  ## kj add 
  coef_dict ['exp'][:] = 0.
  coef_dict['angle'][:]= 0.
  face_shape = self.compute_shape(coef_dict['id'], coef_dict['exp'])
  rotation = self.compute_rotation(coef_dict['angle'])

• As for 7 3d landmarks metioned in Now challenge , I selected points like this:

  recon_shape = self.pred_vertex  # get reconstructed shape
  # print("recon_shape2:",recon_shape)
  # recon_shape[..., -1] = 10 - recon_shape[..., -1] # from camera space to world space  #save mesh part have already transformed shape to  world space 
  recon_shape = recon_shape.cpu().numpy()[0]
  lm_3d_indx = self.facemodel.keypoints.cpu().numpy()[[36,39,42,45,33,48,54]]
  lm_3d = recon_shape[lm_3d_indx,:]

I hope that you could point out the which step is wrong or what i missed , I would really apreciate it. It would be great if you could provide the evaluation code for Now challange.

[YuDeng]

Hi, for NoW challenge, we use the full head region of BFM (which contains ears and neck) instead of the cropped face region. This is a key factor for reaching lower reconstruction error.

[lhyfst]

As YuDeng 's suggestion, what's your result on NoW if you use the full head region? @1180800817

[HITKJ]

Hi, for NoW challenge, we use the full head region of BFM (which contains ears and neck) instead of the cropped face region. This is a key factor for reaching lower reconstruction error.

• Thanks for your reply, I get much lower error on validation part of NoW challenge using the full head region of BFM. (deep3d_pytorch_official_full_region - median: 1.093577, mean: 1.372326, std: 1.158287). (deca_nexp - median: 1.174798, mean: 1.459256, std: 1.244139).

• It's better than deca on validation part and worse than deca on test part of NoW challenge. Is there any reasonable explanation for this ？Is it caused by some attributes of the test datasets?

• It seems that NoW challenge can not comprehensively evaluate the quality of face reconstruction. What evaluation methods do you think are most important for face reconstruction?

[lhyfst]

Did you test the pretrained model on the validation part? Or you trained a new model? @1180800817

[AyushP123]

+1 to @lhyfst question. Is there a pre-trained model for the full head region?? If not then its not possible right to include full head region for NoW benchmark

[HITKJ]

Did you test the pretrained model on the validation part? Or you trained a new model? @1180800817

I use the pretrained model

[AyushP123]

@1180800817 what changes did you make to include the full face region??

[HITKJ]

@AyushP123 You need to change the function transferBFM09() in load_mats.py .it shows how to crop a template face.

[AyushP123]

Thank you for your response @1180800817!!

[zhih-li]

@AyushP123 You need to change the function transferBFM09() in load_mats.py .it shows how to crop a template face.

Thank you for sharing!! Would it be convenient for you to share the details of the code modification? I simply modified index_Shape in the function transferBFM09() , but it didn't seem to work. Did you regenerate the triangle information from original_BFM? Thanks again

[LY-Donkey-Burger]

嗨，对于现在的挑战，我们使用 BFM 的完整头部区域（包含耳朵和颈部）而不是裁剪的面部区域。这是达到较低重构误差的关键因素。

Hi，how should I transfer the cropped face region to full head region of BFM?

[LY-Donkey-Burger]

@AyushP123您需要更改 load_mats.py 中的函数 transferBFM09() 。它显示了如何裁剪模板面​​。

Hi.Would it be convenient for you to share the details of the code modification?

[Alandougherty]

@AyushP123 You need to change the function transferBFM09() in load_mats.py .it shows how to crop a template face.

Thank you for sharing!! Would it be convenient for you to share the details of the code modification? I simply modified index_Shape in the function transferBFM09() , but it didn't seem to work. Did you regenerate the triangle information from original_BFM? Thanks again

I also am having this same problem. Did you manage to sovle it?

[WangXuan2401]

@AyushP123 You need to change the function transferBFM09() in load_mats.py .it shows how to crop a template face.

hello！I'm also struggling with the "full region of BFM" problem. Would you be nice to give me more advice of the code modification？

Thanks for your attention！

[wengjincheng]

@AyushP123 You need to change the function transferBFM09() in load_mats.py .it shows how to crop a template face.

hello！I'm also struggling with the "full region of BFM" problem. Would you be nice to give me more advice of the code modification？

Thanks for your attention！

@WangXuan2401 HI, can you share the method to get the full region?