Feature request to include face rotation in facemesh

[n0noob]

To get help from the community, we encourage using Stack Overflow and the tensorflow.js tag.

TensorFlow.js version

NA

Browser version

NA

Describe the problem or feature request

Currently facemesh detection results include only 468 landmark points. Wouldn't it be better to include face rotation as well in the detection results ?

Code to reproduce the bug / link to feature request

If you would like to get help from the community, we encourage using Stack Overflow and the tensorflow.js tag.

GitHub issues for this repository are tracked in the tfjs union repository.

Please file your issue there, following the guidance in that issue template.

[dikatok]

@n0noob you mean like tilting your head some degree to the left/right? If yes, you can calculate it by yourself. Something like this.

const predictions = await model.estimateFaces(video);

if (predictions.length > 0) {
  const { annotations } = predictions[0];

  const [topX, topY] = annotations['midwayBetweenEyes'][0];

  const [rightX, rightY] = annotations['rightCheek'][0];
  const [leftX, leftY] = annotations['leftCheek'][0];
  const bottomX = (rightX + leftX) / 2;
  const bottomY = (rightY + leftY) / 2;

  const degree = Math.atan((topY - bottomY) / (topX - bottomX));

  console.log(degree);
}

Disclaimer: I didn't test the code above but you can fiddle around with which annotations to use and more precise way to calculate the degree of rotation.

[DamascenoRafael]

I think what he meant was to include the pitch, yaw and roll angles - Aircraft_principal_axes.

It would be an interesting addition and would bring more possibilities for using the model.

[n0noob]

@DamascenoRafael Yes, thank you, this is what I meant.

[rgs2151]

Any idea on how I can calculate the Yaw of the facemesh assuming I was to look at it from the front?

[annxingyuan]

Hi @wakoan - I think we discussed this at some point - and you showed me your team's solution to this problem. Would you mind transferring context to @lina128 ? Thank you!

[akhirai560]

@n0noob Use this code for x and y rotations i am using this for liveness detection:

getHeadAnglesCos(keypoints) { // V: 10, 152; H: 226, 446 this.faceVerticalCentralPoint = [ 0, (keypoints[10][1] + keypoints[152][1]) 0.5, (keypoints[10][2] + keypoints[152][2]) 0.5, ]; const verticalAdjacent = keypoints[10][2] - this.faceVerticalCentralPoint[2]; const verticalOpposite = keypoints[10][1] - this.faceVerticalCentralPoint[1]; const verticalHypotenuse = this.l2Norm([ verticalAdjacent, verticalOpposite, ]); const verticalCos = verticalAdjacent / verticalHypotenuse;

this.faceHorizontalCentralPoint = [
  (keypoints[226][0] + keypoints[446][0]) * 0.5,
  0,
  (keypoints[226][2] + keypoints[446][2]) * 0.5,
];
const horizontalAdjacent =
        keypoints[226][2] - this.faceHorizontalCentralPoint[2];
const horizontalOpposite =
        keypoints[226][0] - this.faceHorizontalCentralPoint[0];
const horizontalHypotenuse = this.l2Norm([
  horizontalAdjacent,
  horizontalOpposite,
]);
const horizontalCos = horizontalAdjacent / horizontalHypotenuse;

return [
  verticalCos,
  horizontalCos,
];

}

l2Norm(vec) { let norm = 0; for (const v of vec) { norm += v * v; } return Math.sqrt(norm); }

[google-ml-butler[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. Thank you.

[DamascenoRafael]

It is still an interesting feature to be implemented

[lina128]

I will follow up with @wakoan on this feature.

[AlexShafir]

@n0noob Are you sure rotation angles are really needed? Maybe rotation matrix 3x3 will be enough?

For face mesh, it is trivial to find rot matrix: there are points on left/right cheeks that define "x", points on bottom of mesh and top that define "y", their cross product is "z". Normalize each vector and R = [ x_n y_n z_n ].

Obtaining Euler angles (roll, pitch, yaw) from rotation matrix is doable, but has disadvantages like gimbal lock.

Sinthanayothin et al. "Morph targets for 3D facial animation ..." (link) use this idea:

[AlexShafir]

@lina128 I have opened corresponding MR in tfjs-models.

[lina128]

Hi @n0noob and @AlexShafir, we are working on providing the rotating feature to face-landmark API. It will likely be launched in January. I'm adding the engineer who will be working on it. In the meantime, feel free to patch the solutions the community provides here.

[AlexShafir]

@lina128 are contributions still welcome or "stat:contributions welcome" label should be removed?

[rthadur]

Facemesh is deprecated , please try our latest face-detection model which has support included for "rotated faces". Thank you

[akssieg]

@rthadur How to access that "rotated faces" feature?

[lina128]

We are still working on an accurate approach to get the transformation for face. Stay tuned.

[gauravmankotia]

any update on this?

[parth7326]

We are still working on an accurate approach to get the transformation for face. Stay tuned.

Any update on this?

[gaikwadrahul8]

Hi, @n0noob

Apologize for the delayed response and @tensorflow-models/facemesh , that package has been archived in favor of @tensorflow-models/face-landmarks-detection and will no longer be updated. you can refer here. Please go with @tensorflow-models/face-landmarks-detection.

If you face any issues with Face Landmarks Detection, Please let us know ? Thank you!

[github-actions[bot]]

This issue has been marked stale because it has no recent activity since 7 days. It will be closed if no further activity occurs. Thank you.

[github-actions[bot]]

This issue was closed due to lack of activity after being marked stale for past 7 days.