Received unknown JNI type "class android.graphics.PointF"! Cannot convert to jsi::Value.

[yiluzhang]

When using the front camera to recognize a face, an error occurs: Frame Processor Error: Exception in HostFunction: Received unknown JNI type "class android.graphics.PointF"! Cannot convert to jsi::Value., js engine: VisionCamera

Normal recognition using back camera.

Huawei HarmonyOS 3.0

"react-native": "0.72.4", "react-native-reanimated": "3.7.0", "react-native-vision-camera": "3.9.0", "react-native-vision-camera-face-detector": "1.2.3", "react-native-worklets-core": "0.3.0"

[luicfrr]

Can you please test face detection disabling face landmarks and face contourns? (Maybe running with default options)

Unfortunatelly I don’t have any device running harmonyOS so I would appreciate if you can help trying to find a fix and opening a PR

[luicfrr]

Android PointF is imported here https://github.com/nonam4/react-native-vision-camera-face-detector/blob/ba68a4ecd8ce83c057fd16132d87f88c2575c43a/android/src/main/java/com/visioncamerafacedetector/VisionCameraFaceDetectorPlugin.java#L5

[yiluzhang]

Can you please test face detection disabling face landmarks and face contourns? (Maybe running with default options)

Unfortunatelly I don’t have any device running harmonyOS so I would appreciate if you can help trying to find a fix and opening a PR

const handleDetectionWorklet = Worklets.createRunInJsFn((result: DetectionResult) => {
  console.log(JSON.stringify(result));
});

const frameProcessor = useFrameProcessor(
  frame => {
    'worklet';
    runAsync(frame, () => {
      'worklet';
      detectFaces(frame, handleDetectionWorklet, {});
    });
  },
  [handleDetectionWorklet],
);

/** Logcat
ImageReader::onImageAvailable!
detectFacesImageByteBufferMultiPlanes.start()
ImageReader::onImageAvailable!
ImageReader::onImageAvailable!
ImageReader::onImageAvailable!
detectFacesImageByteBuffer.end()
{"faces":{"0":{"bounds":{"bottom":1513,"centerX":502,"centerY":1208,"height":610,"left":198,"right":807,"top":903,"width":609},"pitchAngle":-10.01291561126709,"rollAngle":1.969938039779663,"yawAngle":-5.122788906097412}},"frame":{"height":1080,"orientation":"portrait","width":1920}}
 */

/** Sometimes it always returns
{ faces: { '0': undefined }, frame: { height: 1080, orientation: 'portrait', width: 1920 } }
 */

detectFaces(frame, handleDetectionWorklet, {
  performanceMode: 'accurate',
  landmarkMode: 'all',
  contourMode: 'all',
  convertFrame: true,
});
/** Sometimes return error
Error processing face detection: 
java.lang.NullPointerException: Attempt to invoke virtual method 'java.util.List com.google.mlkit.vision.face.FaceContour.getPoints()' on a null object reference
at com.visioncamerafacedetector.VisionCameraFaceDetectorPlugin.processFaceContours(VisionCameraFaceDetectorPlugin.java:183)
  at com.visioncamerafacedetector.VisionCameraFaceDetectorPlugin.callback(VisionCameraFaceDetectorPlugin.java:239)
  at com.facebook.jni.ThreadScopeSupport.runStdFunctionImpl(Native Method)
  at com.facebook.jni.ThreadScopeSupport.runStdFunction(ThreadScopeSupport.java:32)
 */

Hope it helps, for u

[luicfrr]

Can you test modifiying processFaceContours method inside VisionCameraFaceDetectorPlugin.java file, replace whole method with bellow code:

private Map processFaceContours(Face face) {
    int[] faceContoursTypes = new int[] {
      FaceContour.FACE,
      FaceContour.LEFT_CHEEK,
      FaceContour.LEFT_EYE,
      FaceContour.LEFT_EYEBROW_BOTTOM,
      FaceContour.LEFT_EYEBROW_TOP,
      FaceContour.LOWER_LIP_BOTTOM,
      FaceContour.LOWER_LIP_TOP,
      FaceContour.NOSE_BOTTOM,
      FaceContour.NOSE_BRIDGE,
      FaceContour.RIGHT_CHEEK,
      FaceContour.RIGHT_EYE,
      FaceContour.RIGHT_EYEBROW_BOTTOM,
      FaceContour.RIGHT_EYEBROW_TOP,
      FaceContour.UPPER_LIP_BOTTOM,
      FaceContour.UPPER_LIP_TOP
    };

    String[] faceContoursTypesStrings = {
      "FACE",
      "LEFT_CHEEK",
      "LEFT_EYE",
      "LEFT_EYEBROW_BOTTOM",
      "LEFT_EYEBROW_TOP",
      "LOWER_LIP_BOTTOM",
      "LOWER_LIP_TOP",
      "NOSE_BOTTOM",
      "NOSE_BRIDGE",
      "RIGHT_CHEEK",
      "RIGHT_EYE",
      "RIGHT_EYEBROW_BOTTOM",
      "RIGHT_EYEBROW_TOP",
      "UPPER_LIP_BOTTOM",
      "UPPER_LIP_TOP"
    };

    Map<String, Object> faceContoursTypesMap = new HashMap<>();
    for (int i = 0; i < faceContoursTypesStrings.length; i++) {
      FaceContour contour = face.getContour(faceContoursTypes[i]);
      String contourName = faceContoursTypesStrings[i];

      Log.d(TAG, "Getting '" + contourName + "' contour");
      if(contour == null) {
        Log.d(TAG, "Face contour '" + contourName + "' is null - going next");
        continue;
      }

      List<PointF> points = contour.getPoints();
      List<Map<String, Double>> pointsArray = new ArrayList<>();

      for (int j = 0; j < points.size(); j++) {
        Map<String, Double> currentPointsMap = new HashMap<>();
        currentPointsMap.put("x", (double) points.get(j).x);
        currentPointsMap.put("y", (double) points.get(j).y);
        pointsArray.add(currentPointsMap);
      }
      faceContoursTypesMap.put(contourName, pointsArray);
    }

    return faceContoursTypesMap;
  }

See if this solves the error you're recieving. If it does I'll publish a new version with this fix.

[luicfrr]

Reopening issue as I don't know if it's really fixed. @yiluzhang I'm waiting for your feedback.

[yiluzhang]

Reopening issue as I don't know if it's really fixed. @yiluzhang I'm waiting for your feedback.

wow, I tried it and this code works. But when opening landmarkMode, an error occurred:

/**
2024-02-21 22:12:11.913 26027-26140 FaceDetector            com.camera                           D  ------------------ yilu debug processLandmarks ------------------
2024-02-21 22:12:11.913 26027-26140 FaceDetector            com.camera                           D  Getting 'LEFT_CHEEK' landmark
2024-02-21 22:12:11.913 26027-26140 FaceDetector            com.camera                           D  Getting 'LEFT_EAR' landmark
2024-02-21 22:12:11.913 26027-26140 FaceDetector            com.camera                           D  Getting 'LEFT_EYE' landmark
2024-02-21 22:12:11.913 26027-26140 FaceDetector            com.camera                           D  Getting 'MOUTH_BOTTOM' landmark
2024-02-21 22:12:11.913 26027-26140 FaceDetector            com.camera                           D  Getting 'MOUTH_LEFT' landmark
2024-02-21 22:12:11.913 26027-26140 FaceDetector            com.camera                           D  Getting 'MOUTH_RIGHT' landmark
2024-02-21 22:12:11.913 26027-26140 FaceDetector            com.camera                           D  Getting 'NOSE_BASE' landmark
2024-02-21 22:12:11.913 26027-26140 FaceDetector            com.camera                           D  Getting 'RIGHT_CHEEK' landmark
2024-02-21 22:12:11.913 26027-26140 FaceDetector            com.camera                           D  Getting 'RIGHT_EAR' landmark
2024-02-21 22:12:11.913 26027-26140 FaceDetector            com.camera                           D  Getting 'RIGHT_EYE' landmark
2024-02-21 22:12:11.914 26027-26140 VisionCamera            com.camera                           E  Received unknown JNI type "class android.graphics.PointF"! Cannot convert to jsi::Value.
2024-02-21 22:12:11.917 26027-26140 VideoPipeline           com.camera                           I  ImageReader::onImageAvailable!
2024-02-21 22:12:11.917 26027-26141 ReactNativeJS           com.camera                           E  Frame Processor Error: Exception in HostFunction: Received unknown JNI type "class android.graphics.PointF"! Cannot convert to jsi::Value., js engine: VisionCamera
 */

[luicfrr]

Thanks for reply, can you check if 2a9c7a6 fix this last issue for you?

[yiluzhang]

Thanks for reply, can you check if 2a9c7a6 fix this last issue for you?

I tried it on Xiaomi, Huawei, Honor, OPPO, VIVO, and it works. great.

[luicfrr]

Will release a new version with this fix soon