Missing documentation regarding the anatomical definitions of the 468 face landmarks

[fangq]

my student and I are working on a robust registration between face-mesh with other head/brain-landmarks used for neuroimaging, specifically, the 10-20 system for EEG.

the comments in a previous issue (#1615) seem to give a good picture how these face landmarks are numbered, but I still don't see a rigorous definition on where these key points are supposed to be from a face-feature perspective, for example, where is landmark 10 is supposed to be located? middle of the forehead? is there a standard or document where I can find face landmark's "actual" geometric anatomical definitions?

thanks

[kuaashish]

Hi @fangq, Is it implemented through Python, Javascript or other solutions available in MediaPipe? Could you please provide the complete details to understand the issue better from our end. Thank you!

[fangq]

Currently, our initial implementation is based on python, but the question is more generic and independent of implementation.

on a related note - among the 468 face mesh points, are there "dependent" points that are computed/interpolated based on other independently estimated points? if so, what are those and how they are interpolated?

[rajkundu]

I'm not sure if this helps at all, but there is a .obj file located here in the MediaPipe repository. You can open this file in external software to visualize and analyze the face mesh model. For example, using MeshLab, you can get the index of a vertex in a mesh as shown below:

To do so: 1) Open MeshLab, which will create a new, empty project. 2) Drag & drop canoncal_face_model.obj onto the window, which will import the mesh into this new project. 3) Click the "Get Info" button (yellow circle with an "i" in the icon bar). 4) Press "t" to switch from face-selection mode (default) to vertex-selection mode. 5) Click any vertex to get its index.

I am very much a beginner when it comes to MeshLab, but if you have any questions, I would be happy to try and answer them. I recommend playing around with the vertex display size as shown in my screenshot above, on the right, in the sidebar.

[fangq]

@rajkundu thanks for your reply, but I don't think it answers the question I was asking - this still only gives the numbering of points, but not what face feature they are supposed to be pointing to.

just an example - what location on the face that point 10 is supposed to be?

[rajkundu]

Sorry; I'm not entirely sure what you mean. To me, it seems like that question is answerable by looking at the canonical face model UV map, then seeing where Point 10 is. Or, if you're interested in Point 10's 3D definition, I would look at the .obj file.

Perhaps you're looking for an anatomical definition instead?

[fangq]

Perhaps you're looking for an anatomical definition instead?

yes, that is correct. I previously referred it to as "geometric definition", but I think "anatomical definition" is more appropriate.

[rajkundu]

Ah, I see. To be honest, I'm not quite sure how these types of things tend to work. I think the question now is whether Google's engineers attached anatomical definitions to any of the the MP face model's points. I mean, to some extent, strict anatomical correspondences may not even be necessary for the model to function as intended.

My guess is as good as yours, but I would imagine the MP face model does not have many, if any, anatomical correspondences. Some of the only easy/robust ones that I can think of are: the corners of the eyes, the tip of the nose, and maybe even the nasion (vertex 168?). In general, I imagine that finding these correspondences is our job; I don't think there is an official document on them, though I could certainly be wrong.

[fangq]

yes, the only anatomically-clear landmark relevant to my project is nasion (pt168), other than that, I have been computing other 10-20 anchor points empirically by extrapolating 10, 234, 227, 457 and 447, which worked reasonably well, but I was hoping if all landmarks were anatomically defined, then I would get more assets to improve the mapping.

[andrechen]

It always puzzle me to find out the the landmark id. So I injected .obj file with material section. And use a simple material file to bind the uv visualization map. It makes the obj could be opened in Mesh viewer with vertex id...

or you can import canonical face model to blender, then find interpolation weights to define your customized landmarks...

[kuaashish]

Hello @fangq,

Are you still looking for resolution on this Or the issue has been resolved? Thank you!!

[fangq]

@kuaashish, thanks for following up. We are still looking for official documentation on these landmarks. My student had spent the past two semesters trying to fit these facial landmarks to anatomically defined head landmarks using a large library, but if knowing more on their definitions, even a subset, will still help for refining our registration.

[kuaashish]

@fangq,

Thanks for the confirmation, For now, We do not have any anatomical naming scheme for each of the 468 landmark. However, We can mark this as feature request and share internally based on discussion, We will prioritise the request.

[rezinghost]

i'm searching for definitions too, and i just found this keypoints the orders of landmark indexes are mirrored e.g. in rightEyeUpper0 and leftEyeUpper0. But still lack of that on other facial parts like check or forehead.