apple2373
commented
1 year ago I don't believe you can achieve this easily with Mediapipe. Mediapipe is trained to detect 3D landmarks that correspond to the 468 vertices in its predefined mesh (canonical_face_model.obj). To detect 3D landmarks for your own custom face mesh, you would likely need to train another neural network to do so. This means you have to start with preparing training data, etc.
This is what they say about how they trained the model.
For 3D face landmarks we employed transfer learning and trained a network with several objectives: the network simultaneously predicts 3D landmark coordinates on synthetic rendered data and 2D semantic contours on annotated real-world data. The resulting network provided us with reasonable 3D landmark predictions not just on synthetic but also on real-world data. (source: https://github.com/google/mediapipe/blob/e7121e4feb986bc067c879c954051f8c6cf1eaa2/docs/solutions/face_mesh.md)
They used synthetic data so you may be able to do something similar if you can render realistic faces of your custom mesh. You can also check their paper for details: https://arxiv.org/abs/1907.06724 .
in code you use canonical_face_model.obj(mediapipe defined), if i want to use my custom face mesh, how to do it by mediapipe, thank you