search

luicfrr / react-native-vision-camera-face-detector

Vision Camera Frame Processor Plugin to detect faces using MLKit Face Detector
https://www.npmjs.com/package/react-native-vision-camera-face-detector
MIT License
75 stars 16 forks source link

Face Boundaries shouldn't use Scale #35

Closed mrousavy closed 2 months ago

mrousavy commented 2 months ago

Hey @nonam4, I'm creating this issue to follow up on the discussion about returned coordiantes/bounds being incorrect.

A Camera streams in it's native resolution, and sometimes in landscape orientation, since that's just how sensors work. If I run any ML model on such a native raw frame, I expect the coordinates to be relative to exactly the frame dimensions and orientation, not the preview/screen size or orientation.

This is especially relevant when I want to draw back to the Frame (with Skia Frame Processors), because then the coordinate system will be the frame's .width and .height.

I am talking about these two lines: https://github.com/nonam4/react-native-vision-camera-face-detector/blob/22c87fe4af1fd44fdff7bf729dd9bebe41da176e/ios/VisionCameraFaceDetector.swift#L245-L246

Currently when drawing a blur mask over the face using Skia Frame Processors, it looks like this:

https://github.com/nonam4/react-native-vision-camera-face-detector/assets/15199031/0df58112-316e-4eb5-8b2b-201eee5042e0

After I change scaleX and scaleY to 1.0, it will look correct:

https://github.com/nonam4/react-native-vision-camera-face-detector/assets/15199031/d0009589-1506-4f76-8d9d-ad7b4ecde0f5

Can you remove scaleX/scaleY from your code, and maybe provide such converters on the JS side using the Dimensions.get('window') APIs from React Native? I think that makes more sense, especially now that people use Skia Frame Processors.

luicfrr commented 2 months ago

@mrousavy can you please check v1.5 branch? I added a condition auto scalling prop on it so you can handle scaling on js side.

I'm still testing this branch before merging but I'm having a problem on IOS that makes my frame processor freeze after a few detections.

I think this problem is related to this issue but I'm still working on it.

mrousavy commented 2 months ago

Did you try with latest V4? I am currently running it and everything's working great

mrousavy commented 2 months ago

I just created a demo app; check out https://github.com/mrousavy/FaceBlurApp

You can reproduce the issue there. With scaleX/scaleY set to 1.0, it all looks correct. With your values, it seems to be a bit off.

luicfrr commented 2 months ago

Did you try with latest V4? I am currently running it and everything's working great

it was just me messing with frame 😅

luicfrr commented 2 months ago

@mrousavy In the new version 1.6.0 you now have the option to control whether the plugin handles scaling or not 🚀