facemesh-js:how to use the landmark-z value under Three.js perspective camera?

[tomleader]

We know that: send a image/video to facemesh, then we can get an key point landmarks object array like[{x:0,y:0,z:0},...] sometimes we use Three.js to build some 3d or AR object around the face(which means the object's position&scale == face's position & scale), I can use the x&y to calculate the 3d position (x,y) but the z value I can not figure out how to use. The short question is ,input a landmark object array, a perspective camera, how can I get the 3d object (x,y,z) & scale, is there some examples?

[tomleader]

@sgowroji ok,I create a codepen demo(under chrome) : https://codepen.io/tomleader/pen/XWMaZqr

allow webcam permission, then click the top-right button. Then you can see a box in the center of canvas and you move your head and the box also move by your motion;

how the demo run:

1. init FaceMesh,webcam and threejs(camera, scene, cube);
2. run loop: send video to FaceMesh and get landmarks, in this simple demo, I use the nose position to 'map' the box position; it's some Math job: use the landmark xy value to calculate the box's xy value;
3. You'll see, if my head move to the left and the box also moves to the left, if I move up and the box goes up....

Question: How to set the right size of the box? (eg, if I come closer to the webcam, my head & the box should looks bigger, when I move away the box should look smaller). In threejs, we can set the box size by : 1. set the perspective camera & object's z-position value; 2. set the object's scale value;

[tyrmullen]

This sounds like a question about the various face transforms used, in this case the one which maps 3d (x,y,z) landmark points to world-space (or something approximating a 3d world-space, but with relative scaling) coordinates. The expert on these sorts of transforms is kostyaby@, so reassigning to him.

[kostyaby]

Hey @tomleader,

What you need is Face Geometry module for JS, which currently not supported (please see #1457 for details). It'll be available eventually; however, unfortunately, I don't think this is a top priority for the MediaPipe Team at the moment.

In the meantime, you may give a go to the library suggested in this comment; it seems to be placing a mesh inside a THREE.PerspectiveCamera quite nicely

[tomleader]

@kostyaby thanks but that library use THREE.OrthographicCamera...I think I'll wait for the Face Geometry module for js.

[kostyaby]

Hmm, interesting. I could find a PerspectiveCamera in this example. Maybe its support is limited, but that example definitely works somehow :)

[google-ml-butler[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you.

[google-ml-butler[bot]]

Closing as stale. Please reopen if you'd like to work on this further.

[google-ml-butler[bot]]

Are you satisfied with the resolution of your issue? Yes No

[JpEncausse]

@tomleader did you find a work around ? Can you post an updated codepen ? I'd love to use MediaPipe and ThreeJS (or BabylonJS)

• for holographic effect on a box like you did
• to display glasses and other 3D object on top of face Thanks !

[JpEncausse]

Yes, I correctly import your code in my usecase but I agree for this "paralax effect" it's better to have a static cube and a dynamic camera that retrieve it's corrdinte from MediaPipe landmarks. But I don't know how t odo that :-)

it seems @vivien000 did it but the code is way more complexe https://github.com/vivien000/trompeloeil

[dodorian92]

Hi everyone. Are there any solutions in JS today to use face mesh points detected by mediapipe for placing 3D models on the face? I'm working on a project where I'm using mediapipe in react. I was able to successfully render my Glb 3D model on the face but I would like to try to place it in a specific point of the detected mesh. I also use a second model to simulate the occlusion of the face and this seems to fit the face perfectly. How can I find the coordinates of this point?

[dagon10102010]

in threejs camera fov is vertical fov so zbox = screenheight0.5/Math.tan(camera.fov0.5*Math.PI/180)