FaceMesh Triangles

[shiffman]

Following up on our meeting from this morning, we discussed adding a constant with a pre-defined list of triangles to the ml5.FaceMesh class. I realized afterwards this is similar to the getSkeleton() function in ml5.BodyPose. One idea would then be:

let faceMesh = ml5.faceMesh();
let triangles = faceMesh.getTriangles();

or

let triangles = faceMesh.getTriangulation();

But since there is only one set of triangles (rather than different ones based on MoveNet vs. BlazePose, it doesn't necessarily require a function. It could be a static constant:

let triangles = ml5.faceMesh.TRIANGLES

or an instance constant (?)

let faceMesh = ml5.faceMesh();
let triangles = faceMesh.TRIANGLES;

I think the static way would be more conventional JS?

Here is my example that uses the triangles which is based off of @jackbdu's teaching. Jack, just curious, where did you find this list?

https://editor.p5js.org/ima_ml/sketches/hyxD1BVVn

We could also consider reformatting the triangles like so:

const TRIANGLES = [
  { a: 127, b: 34, c: 139 },
  { a: 11, b: 0, c: 37 },
  { a: 232, b: 231, c: 120},
  ...
}

The current format is:

const TRIANGULATION = [
    127, 34, 139, 11, 0, 37, 232, 231, 120,...

Also, TRIANGLES or TRIANGULATION? I think I prefer TRIANGLES?

@MOQN and @gohai would love to hear your thoughts along with everyone else!

[gohai]

I had also very much forgotten about the precedent in ml5.BodyPose..! 😅

No strong feelings one way or another, but two thoughts came to mind:

• MESH as another possible name
• Since ml5.faceMesh is a function returning a new instance, I am not sure if there is a way to also expose a constant like ml5.faceMesh.TRIANGLES through it?

[shiffman]

Ah, great point @gohai! A static variable would have to be referenced from ml5.FaceMesh and I think that may be especially confusing since we use the "factory" method ml5.faceMesh() as the primary way to create a model instance. I'm leaning now towards "future proofing" the library (what if we have more faceMesh models in the future!) and following the getSkeleton() pattern. Does anyone prefer one of these two options?

• getTriangles()
• getTriangulation()

[jackbdu]

@shiffman I love this example sketch you created!

I found the list in this ar-face-filters repo, which was a project featured in a TensorFlow video and included in @yining1023's fall 2023 syllabus.

I think future proofing is a good strategy! I'm leaning more towards getTriangles() because of its approachability. I'm also considering something like getMesh(), following @gohai's suggestion. The latter feels more consistent with getSkeleton() and more semantic, but could be potentially confusing to a beginner.

In addition, maybe we can also consider a format like this following the getSkeleton() method?

[
  [127, 34, 139],
  [11, 0, 37],
  [232, 231, 120],
  ...
]

[shiffman]

Thank you @jackbdu! I love the the nested arrays since that follows the getSkeleton() pattern! I suppose I could make an argument for an array of objects over an array of arrays, but let's go with arrays to be consistent for these.

I think I prefer getTriangles(). A mesh could technically imply other structures, like quadrilaterals or polygons (?) beyond just triangles, and since it’s also the term for the entire model output itself, I think offering the specificity of what you are getting is best.

I hope to record a faceMesh() tutorial early next week so we can take a little time to discuss further and maybe do a release with this + #219 from @alanvww later this week! cc @sharellb and @ziyuan-linn

[ziyuan-linn]

I also prefer faceMesh.getTriangles()!

Looks like tfjs's face-land-mark detector has a built-in method that returns the connected pairs of key points. The returned array looks like this:

[
  [127, 34],
  [34, 139],
  [139, 127],
  [11, 0],
  [0, 37],
  [37, 11],
  // ...
]

This array can be easily transformed into a triangle array. On another note, would it also be helpful provide the pairs array in addition to the triangles array? Perhaps we can call it something like faceMesh.getConnections()?

I can work on this today and perhaps we can do a release tomorrow or Monday? @shiffman @sharellb

[shiffman]

Oh I like the idea of faceMesh.getConnections()! If we do this though, I wonder if we should deprecate getSkeleton() and rename BodyPose to use getConnections() to be more consistent. And maybe we should do one for handPose as well?

I am also working on another example based on @jackbdu's work that maps a mask image texture to faceMesh with uv points. While it is tailored for specific mask images (see Jack's demo , it works nicely with any abstract image. This is the repo where the uv points came from. Does anyone know if this is a "standard" uv mapping across a variety of face models? I have a vague memory of @golanlevin teaching something along these lines?

My example is here: https://editor.p5js.org/codingtrain/sketches/LSPrCMfnn

This is probably beyond the scope of what we would offer maybe I just keept it as a separate example... but we could consider a faceMesh.getTextureCoordinates() or something along those lines?

[shiffman]

I've been doing some more research on this and found this very helpful resource: https://github.com/lschmelzeisen/understanding-mediapipe-facemesh-output?tab=readme-ov-file

I'm going to do some tests with these uv coordinates and if it works out well, we may want to add this into ml5 as well:

https://github.com/tensorflow/tfjs-models/blob/838611c02f51159afdd77469ce67f0e26b7bbb23/face-landmarks-detection/src/mediapipe-facemesh/uv_coords.ts

[shiffman]

I've done some tests now with these uv coordinates provided in the tfjs documentation. I can confirm they match up exactly. On the left is the faceMesh diagram and the right is that diagram texture mapped with blue dots drawn over it at the faceMesh points.

This allows me to do things like draw my own "mask" elements (left) using the faceMesh diagram as a guide and texture map it with ml5.js.

I can use any image:

Here is the API I propose:

• getTriangles(): returns vertex indices for triangles in the form of [[a,b,c], [a, b, c], ... ] as in #222
• getConnections(): returns vertex indices for the "edges" in the form of [[a,b], [a, b], ... ]
• getUVCoords(): returns UV coordinates for each keypoint in the form of [[u,v], [u,v], ...]

I don't mean to rush this but I would like to record a video tutorial on this tomorrow, October 22nd. @ziyuan-linn would you like to implement this in a branch or would you prefer I do?

Does anyone have any comments or feedback before I proceed? cc @MOQN @gohai

I'm open to other naming ideas besides getUVCoords(), could be getTextureCoords() or getTextureMapping()?

[ziyuan-linn]

@shiffman I opened a PR to add the getConnections() and getUVCoords() functions. I had to copy and paste the UV coords file into the ml5 codebase since that file no longer exists in the current version of tfjs face-landmark-detection.

I haven't had the chance to test the functions with an example yet, but they are returning the correct values.

Please feel free to make any changes if needed.

[ziyuan-linn]

@shiffman I just refactored and added your example to the PR. Looks like everything is working! Feel free to edit the example as well!

During testing, I noticed some triangular artifacts when I turned my head to the side. This happens in both your original example and the refactored example. It looks like the back faces are being displayed?

I am not sure whether this is supposed to happen or not. Perhaps @jackbdu also has some insights?

[shiffman]

I had to copy and paste the UV coords file into the ml5 codebase since that file no longer exists in the current version of tfjs face-landmark-detection.

Yes, it's strange that this mapping seemed to be part of the repo but no longer is there? That said, it works so well I think it's worth incorporating and we can always update later if we have a better mapping. One issue is that all of these are only compatible with the refineLandmarks: false, if you set it to true then there are 478 keypoints instead of 468. So maybe we need to log a warning or something until we have data for those.

During testing, I noticed some triangular artifacts when I turned my head to the side. This happens in both your original example and the refactored example. It looks like the back faces are being displayed?

Yes, I noticed this as well, I should probably try using the z values as well, I think something is probably happening when the 2D triangles overlap? Regardless it's a rendering issue so I don't think it's something we have to sort out in ml5.js (though would be good to consider for refining the examples.)

[jackbdu]

Woohoo, so glad we found the official UV coordinates that matches the faceMesh diagram!

@shiffman you are right, using z values should fix the artifacts. I had the same artifacts when I only used x and y.

[shiffman]

Video recording complete! These are the raw examples if you want to get a sense on what I demonstrated, it'll be a long video covering a variety of techniques! @jackbdu I reference your lips contour + sound example I'll be in touch about permissions to include a clip!

https://editor.p5js.org/codingtrain/sketches/DGEuFKf87 https://editor.p5js.org/codingtrain/sketches/EjIrb89WY https://editor.p5js.org/codingtrain/sketches/zUKp9n4MW

I didn't include this in the video as I'm doing this without consent/permission to use the likeness, but I was curious to learn about how to map one face onto another. See if you can guess who:

https://editor.p5js.org/codingtrain/sketches/DCmrGMkzL