spite / FaceMeshFaceGeometry

FaceMeshFaceGeometry for FaceMesh
MIT License
405 stars 63 forks source link

Update to mediapipe facemesh #10

Open jthawme opened 2 years ago

jthawme commented 2 years ago

As it looks the tfjs facemesh model is deprecated, do you know of any idea of how to use this with MediaPipe's facemesh instead?

https://google.github.io/mediapipe/solutions/face_mesh.html

Any guidance would be appreciated, thanks so much

jthawme commented 2 years ago

I wouldn't feel confident enough for a PR on this, but to drop some further information, it looks very similar in a lot of ways, but the results from the facemesh library linked above come in like

function onResults(results) {
  const {multiFaceLandmarks} = results;

  // [
  //   ...468 items
  //   { x: 0.4, y: 0.6, z: -0.1 }
  // ]
}

They're all values from 0-1, except z I believe

Atm i am exploring using it by doing

geo.update({
  scaledMesh: multiFaceLandmarks[0].map(vec => Object.values(vec))
})

But its not 100% correct yet, but works without erroring and does create a mesh