Shape's jaw fit abnormally

[Allen-lz]

This work is wonderful!! And I also paid attention to your previous related work https://github.com/tencent-ailab/hifi3dface, I directly replaced hifi3dface/3DMM/files/HIFI3D++.mat with hifi3dface/3DMM/files/HIFI3D++.mat The following error occurred when running: f_sigma0 = (paras) / sigma # (500, 1) ValueError: operands could not be broadcast together with shapes (526,1) (500,1) Then I solved the above error using the following: In optimization/rgbd/step3_prefit_shape.py --> AI-NEXT-Shape datas = h5py.File(os.path.join(modle_base, "shape_ev.mat"), "r") ev_f = np.asarray(datas.get("ev_f")).reshape(-1, 1) sigma_shape = np.sqrt(ev_f / np.sum(ev_f))

--> HIFI3D++ sigma_shape = np.sqrt(basis3dmm['EVs']/np.sum(basis3dmm['EVs'])) # (526, 1)

In hifi3dface/optimization/rgbd/RGBD_utils/AddHeadTool.py --> AI-NEXT-Shape ev_f = pca_info_h["ev_f"] sigma_shape = np.sqrt(ev_f / np.sum(ev_f))

--> HIFI3D++ basis3dmm = scipy.io.loadmat("3DMM/files/HIFI3D++.mat") sigma_shape = np.sqrt(basis3dmm['EVs'] / np.sum(basis3dmm['EVs']))

But the result is much worse:

Am I using HIFI3D++.mat incorrectly?

[haoxianzGit]

Hello，thanks for your interest. Maybe the wrong 3D keypoints are used when calculating ‘trans_base_2_camera’ (function 'get_trans_base_to_camera') in step2*.py. You can test the following 3D keypoints: mu_shape = np.reshape(np.float32(basis3dmm['mu_shape']), [-1,3]) mu_shape_kp = mu_shape[basis3dmm['keypoints'][0]] Please check if the results of step2 and step3 are normal.

[Allen-lz]

Hello，thanks for your interest. Maybe the wrong 3D keypoints are used when calculating ‘trans_base_2_camera’ (function 'get_trans_base_to_camera') in step2*.py. You can test the following 3D keypoints: mu_shape = np.reshape(np.float32(basis3dmm['mu_shape']), [-1,3]) mu_shape_kp = mu_shape[basis3dmm['keypoints'][0]] Please check if the results of step2 and step3 are normal.

thanks for the tip I visualized np.reshape(np.float32(basis3dmm['mu_shape']), [-1, 3]) From the visual result, there is no problem.

I guess the problem might be mu_shape[basis3dmm['keypoints'][0]] what's its role Is it related to a1, a2, a3 in get_trans_base_to_camera?

[yikai28]

The same concern here, the result become worse after using HIFI3D++.... May we ask how to solve it><?