luicfrr / react-native-vision-camera-face-detector

Vision Camera Frame Processor Plugin to detect faces using MLKit Face Detector
https://www.npmjs.com/package/react-native-vision-camera-face-detector
MIT License
77 stars 15 forks source link

How to render face landmarks #12

Closed itlockit2 closed 4 months ago

itlockit2 commented 4 months ago

Firstly, I would like to express my appreciation for the excellent library; it has been instrumental in my project. I have successfully used the example code provided to render bounding boxes for detected faces within my application. However, I am encountering difficulties when trying to render facial landmarks on the screen.

I am using the calcFacePosition function to calculate the positions of the bounding boxes based on the face detection output. This part works as expected. However, when attempting to apply similar logic to render facial landmarks, I am unsure how to correctly adjust the coordinates to fit the orientation and scale of the display.

Here is the function I use to calculate the face position:

function calcFacePosition(bounds: Bounds, frame: FrameData): FacePosType {
    const orientation = (() => {
      switch (frame.orientation) {
        case 'portrait':
          return 0;
        case 'landscape-left':
          return 90;
        case 'portrait-upside-down':
          return 180;
        case 'landscape-right':
          return 270;
      }
    })();
    const degrees = (orientation - 90 + 360) % 360;
    let scaleX = 0;
    let scaleY = 0;

    if (!isIos && (degrees === 90 || degrees === 270)) {
      scaleX = windowWidth / frame.height;
      scaleY = windowHeight / frame.width;
    } else {
      scaleX = windowWidth / frame.width;
      scaleY = windowHeight / frame.height;
    }

    const faceW = bounds.width * scaleX;
    const faceH = bounds.height * scaleY;
    const faceX = (() => {
      const xPos = bounds.left * scaleX;
      if (isIos) {
        return xPos;
      }
      return windowWidth - (xPos + faceW); // invert X position on android
    })();

    return {
      faceW,
      faceH,
      faceX,
      faceY: bounds.top * scaleY,
    };
}

I would appreciate guidance on how to modify this function or apply a different method to accurately position facial landmarks on the screen, taking into account different device orientations and scales.

Thank you for your time and assistance.

luicfrr commented 4 months ago

Hello,

You'll need to loop through all landmarks (x and y values) and apply scale to them. At this moment I can't help you with some code because landmarks are not something I use in my own projects.

However, I can provide you this assistance with a financial compensation. If you're willing to consider this option, I'm here to help in the best way I can.

I'll close this issue as it's not realated to any issue nor a problem but you can still reply it to continue this conversation.