How ./data/landmark_3d.pkl is obtained

[jxcodetw]

I tried to use the 3d landmark produced by https://github.com/1adrianb/face-alignment (Mentioned in https://github.com/TimoBolkart/TF_FLAME) And I manually translate the 68 landmarks to the 49 landmarks

The result is:

Can you give an example how to get the result like ./data/landmark_3d.pkl ?

[TimoBolkart]

Can you give more details on what you did? Fitting to 3D landmarks? Fitting to 2D landmarks? We obtained the landmark_3d.pkl by manually selecting the landmarks on a 3D scan

[jxcodetw]

Hi, I tried to fitting to 3D landmarks that is estimated from a 2D image. The 3D reconstruction is obtained from this tool https://github.com/1adrianb/face-alignment The tool produces 68 landmarks and I pick the 49 landmarks from them based on the reference image and put them into the correct order. https://github.com/Rubikplayer/flame-fitting/blob/master/data/landmarks_order.jpg

[TimoBolkart]

Can you provide the 3D landmarks that you used to fit to, together with your FLAME fitting result? Which code did you use? The Chumpy or the TF code to do the 3D landmark fitting?

In general we do not advice to use predicted 3D landmarks as they represent an estimated depth rather than true 3D coordinates. This can make fitting quite inaccurate.

[kj96]

I am getting similar results.

Landmark genrated from face_alignment (I selected req. 51): array([[253. , 293. , 39.201477], [270. , 276. , 63.531372], [298. , 271. , 79.31812 ], [320. , 271. , 88.14569 ], [342. , 276. , 90.56665 ], [436. , 271. , 87.56543 ], [458. , 265. , 83.627914], [486. , 260. , 73.03325 ], [508. , 271. , 55.706696], [530. , 287. , 29.485266], [392. , 331. , 84.57093 ], [392. , 359. , 99.125656], [392. , 387. , 117.71449 ], [392. , 414. , 118.954796], [364. , 431. , 76.0769 ], [375. , 437. , 82.46238 ], [392. , 437. , 85.349785], [408. , 437. , 81.53199 ], [425. , 431. , 74.68625 ], [287. , 331. , 53.45837 ], [303. , 326. , 70.48598 ], [320. , 326. , 70.38919 ], [342. , 331. , 62.27365 ], [325. , 337. , 68.69526 ], [303. , 343. , 65.53892 ], [441. , 331. , 60.24563 ], [464. , 320. , 66.554375], [480. , 320. , 64.75552 ], [497. , 331. , 45.88586 ], [480. , 337. , 59.164528], [458. , 337. , 64.620415], [331. , 486. , 42.795418], [353. , 481. , 68.81434 ], [381. , 470. , 84.12075 ], [392. , 475. , 85.89977 ], [408. , 470. , 84.29676 ], [436. , 475. , 68.44957 ], [458. , 486. , 40.87157 ], [436. , 503. , 61.537197], [414. , 508. , 72.602005], [392. , 514. , 75.492195], [375. , 514. , 72.97415 ], [353. , 503. , 62.474445], [331. , 486. , 42.76171 ], [375. , 486. , 72.801796], [392. , 486. , 77.12267 ], [414. , 486. , 72.8894 ], [453. , 486. , 41.055923], [414. , 492. , 70.293724], [392. , 492. , 73.13472 ], [375. , 492. , 70.23644 ]], dtype=float32)

This is because landmark generated from face_alignmet are according to input image height, width, But values of landmark used in Flame is much smaller. Here is similar case in TF_FLAME issue15

How should values be normalize?

[plmsmile]

I met a similar problem. I use my photo image and picked 2d landmark values from face-alignment.

def test_landmark():
    img_path = "./data/mf.png"
    key = img_path.rsplit("/", 1)[-1].rsplit(".", 1)[0]
    npy_path = "./data/{}_lmks.npy".format(key)
    fa = face_alignment.FaceAlignment(face_alignment.LandmarksType._2D, flip_input=False, device="cpu")
    img = io.imread(img_path)
    preds = fa.get_landmarks(img)
    preds = preds[1][:51]
    np.save(npy_path, preds)
    return

[TimoBolkart]

I strongly recommend fitting to 2D landmarks rather than estimated 3D landmarks. We have some code available that demonstrates how to fit FLAME to 2D landmarks here TF_FLAME. For the 3D landmark fitting demo, the landmarks must be in the same unit as FLAME, which is provided in Meters. An alternative can be to optimize for an isotropic scale as well.

[JohnFive5]

Thank you for your excellent code!I wonder why you strongly recommend fitting to 2D landmarks but implemented a 3D landmark loss.I tried to implement a 2D loss,using the camera calibration as you suggested in the paper,and transformed the 3D coordinates to pixel coordinates to make a loss function.But it seems that Chumpy can not get the gradient.What's the problem?How should I modify x0？