search

luicfrr / react-native-vision-camera-face-detector

Vision Camera Frame Processor Plugin to detect faces using MLKit Face Detector
https://www.npmjs.com/package/react-native-vision-camera-face-detector
MIT License
75 stars 16 forks source link

[BUG] Inconsistent face detection #22

Closed divyaprakash0426 closed 2 months ago

divyaprakash0426 commented 3 months ago

Bug Description When using the face detection feature of the Vision Camera library in my custom project on an Expo bare workflow, the detection is inconsistent. Sometimes it detects faces accurately, but often it fails to detect any faces at all, as evident from the terminal logs showing intermittent detection results.

To Reproduce Steps to reproduce the behavior:

  1. Set up a project using Expo bare workflow.
  2. Integrate Vision Camera and react-native-vision-camera-face-detector following the example provided.
  3. Run the app on a Pixel 7 Pro device.
  4. Observe the face detection functionality; it intermittently fails to detect faces.

Expected behavior The face detection should consistently detect faces without intermittent failures.

Logs and Screenshots Logs showing intermittent face detection:

LOG faces {} LOG faces {} LOG faces {} LOG faces {} LOG faces {} LOG faces {} LOG faces {} LOG faces {} LOG faces {} LOG faces {} LOG faces {} LOG faces {} LOG faces {} LOG faces {} LOG faces {} LOG faces {} LOG faces {"0": {"bounds": {"height": 307.77708333333334, "width": 155.5, "x": 137.5, "y": 372.04375}, "leftEyeOpenProbability": 0.679590106010437, "pitchAngle": -6.849165439605713, "rightEyeOpenProbability": 0.8706389665603638, "rollAngle": 53.27714157104492, "smilingProbability": 0.004994028247892857, "yawAngle": 5.326096057891846}} LOG faces {"0": {"bounds": {"height": 366.5208333333333, "width": 185.5, "x": 103, "y": 291.2083333333333}, "leftEyeOpenProbability": 0.9942908883094788, "pitchAngle": 11.596040725708008, "rightEyeOpenProbability": 0.9922314882278442, "rollAngle": -96.88375091552734, "smilingProbability": 0.014574314467608929, "yawAngle": -3.906331777572632}} LOG faces {} LOG faces {} LOG faces {} LOG faces {"0": {"bounds": {"height": 359.99375, "width": 179.25, "x": 99.75, "y": 264.0958333333333}, "leftEyeOpenProbability": 0.9979193210601807, "pitchAngle": 8.968412399291992, "rightEyeOpenProbability": 0.9988862872123718, "rollAngle": -93.2209701538086, "smilingProbability": 0.009467480704188347, "yawAngle": -5.748469829559326}} LOG faces {} LOG faces {} LOG faces {} LOG faces {} LOG faces {"0": {"bounds": {"height": 372.54583333333335, "width": 185.5, "x": 85.75, "y": 248.53125}, "leftEyeOpenProbability": 0.9928528666496277, "pitchAngle": 8.301859855651855, "rightEyeOpenProbability": 0.997727632522583, "rollAngle": -88.34501647949219, "smilingProbability": 0.008996601216495037, "yawAngle": -8.334175109863281}} LOG faces {} LOG faces {} LOG faces {} LOG faces {} LOG faces {} LOG faces {"0": {"bounds": {"height": 327.35833333333335, "width": 168.5, "x": 41.25, "y": 356.4791666666667}, "leftEyeOpenProbability": 0.8463578224182129, "pitchAngle": -12.933387756347656, "rightEyeOpenProbability": 0.8650642037391663, "rollAngle": 60.45033264160156, "smilingProbability": 0.02757435292005539, "yawAngle": 4.451534748077393}} LOG faces {} LOG faces {} LOG faces {} LOG faces {} LOG faces {} LOG faces {} LOG faces {} LOG faces {} LOG faces {} LOG faces {"0": {"bounds": {"height": 297.23333333333335, "width": 148, "x": 17.5, "y": 407.1895833333333}, "leftEyeOpenProbability": 0.9022865295410156, "pitchAngle": 23.50724220275879, "rightEyeOpenProbability": 0.29507261514663696, "rollAngle": 88.87084197998047, "smilingProbability": 0.13751231133937836, "yawAngle": 28.76015281677246}} LOG faces {} LOG faces {} LOG faces {} LOG faces {} LOG faces {} LOG faces {} LOG faces {} LOG faces {} LOG faces {} LOG faces {} LOG faces {} LOG faces {} LOG faces {"0": {"bounds": {"height": 317.31666666666666, "width": 155.5, "x": 17.25, "y": 349.95208333333335}, "leftEyeOpenProbability": 0.9022865295410156, "pitchAngle": 14.430815696716309, "rightEyeOpenProbability": 0.042649805545806885, "rollAngle": 98.21409606933594, "smilingProbability": 0.13751231133937836, "yawAngle": 28.81414794921875}} LOG faces {} LOG faces {} LOG faces {} LOG faces {} LOG faces {} LOG faces {} LOG faces {} LOG faces {} LOG faces {} LOG faces {} LOG faces {} LOG faces {} LOG faces {} LOG faces {} LOG faces {} LOG faces {} LOG faces {} LOG faces {} LOG faces {} LOG faces {} LOG faces {} LOG faces {} LOG faces {} LOG faces {} LOG faces {"0": {"bounds": {"height": 343.9270833333333, "width": 168.5, "x": 9.25, "y": 243.51041666666666}, "leftEyeOpenProbability": 0.27911096811294556, "pitchAngle": 4.311995029449463, "rightEyeOpenProbability": 0.6025388240814209, "rollAngle": 64.7170181274414, "smilingProbability": 0.010426056571304798, "yawAngle": 54.951534271240234}} LOG faces {} LOG faces {} LOG faces {} LOG faces {} LOG faces {} LOG faces {} LOG faces {} LOG faces {"0": {"bounds": {"height": 698.3979166666667, "width": 337, "x": 140.75, "y": 412.2104166666667}, "leftEyeOpenProbability": 0.03194314241409302, "pitchAngle": -12.541540145874023, "rightEyeOpenProbability": 0.15379184484481812, "rollAngle": 21.207067489624023, "smilingProbability": 0.31375253200531006, "yawAngle": -0.34272608160972595}} LOG faces {} LOG faces {} LOG faces {"0": {"bounds": {"height": 698.3979166666667, "width": 342.25, "x": 141, "y": 412.2104166666667}, "leftEyeOpenProbability": 0.012776970863342285, "pitchAngle": -13.734760284423828, "rightEyeOpenProbability": 0.6025388240814209, "rollAngle": 22.649320602416992, "smilingProbability": 0.3273563086986542, "yawAngle": 5.618047714233398}} LOG faces {"0": {"bounds": {"height": 666.2645833333333, "width": 336.75, "x": 146.5, "y": 434.3020833333333}, "leftEyeOpenProbability": 0.05539274215698242, "pitchAngle": -12.883161544799805, "rightEyeOpenProbability": 0.632347822189331, "rollAngle": 21.856197357177734, "smilingProbability": 0.3961668908596039, "yawAngle": 8.105184555053711}} LOG faces {"0": {"bounds": {"height": 676.8083333333333, "width": 336.75, "x": 143.75, "y": 428.77916666666664}, "leftEyeOpenProbability": 0.05671572685241699, "pitchAngle": 0.5525782108306885, "rightEyeOpenProbability": 0.8734263181686401, "rollAngle": 19.077682495117188, "smilingProbability": 0.1818937361240387, "yawAngle": 24.352909088134766}} LOG faces {} LOG faces {} LOG faces {} LOG faces {} LOG faces {} LOG faces {} LOG faces {} LOG faces {} LOG faces {} LOG faces {} LOG faces {} LOG faces {} LOG faces {} LOG faces {} LOG faces {} LOG faces {} LOG faces {} LOG faces {} LOG faces {} LOG faces {} LOG faces {} LOG faces {} LOG faces {} LOG faces {} LOG faces {} LOG faces {} LOG faces {} LOG faces {} LOG faces {} LOG faces {} LOG faces {} LOG faces {} LOG faces {} LOG faces {} LOG faces {} LOG faces {} LOG faces {} LOG faces {} LOG faces {} LOG faces {} LOG faces {} LOG faces {} LOG faces {} LOG faces {} LOG faces {} LOG faces {} LOG faces {"0": {"bounds": {"height": 325.35, "width": 164.75, "x": 163, "y": 212.88333333333333}, "leftEyeOpenProbability": 0.1954095959663391, "pitchAngle": 10.456294059753418, "rightEyeOpenProbability": 0.6963686943054199, "rollAngle": -94.03935241699219, "smilingProbability": 0.0058485232293605804, "yawAngle": -43.203369140625}} LOG faces {"0": {"bounds": {"height": 319.82708333333335, "width": 156.5, "x": 136, "y": 354.97291666666666}, "leftEyeOpenProbability": 0.7634485363960266, "pitchAngle": 4.588699817657471, "rightEyeOpenProbability": 0.9383997917175293, "rollAngle": -33.952857971191406, "smilingProbability": 0.011907465755939484, "yawAngle": 39.479976654052734}} LOG faces {} LOG faces {} LOG faces {} LOG faces {} LOG faces {} LOG faces {} LOG faces {} LOG faces {} LOG faces {} LOG faces {} LOG faces {} LOG faces {} LOG faces {} LOG faces {} LOG faces {} LOG faces {} LOG faces {} LOG faces {} LOG faces {} LOG faces {} LOG faces {} LOG faces {} LOG faces {} LOG faces {} LOG faces {} LOG faces {} LOG faces {"0": {"bounds": {"height": 360.49583333333334, "width": 182.25, "x": 181.75, "y": 400.16041666666666}, "leftEyeOpenProbability": 0.9868497848510742, "pitchAngle": 8.496673583984375, "rightEyeOpenProbability": 0.9841023683547974, "rollAngle": -87.61302185058594, "smilingProbability": 0.01140771247446537, "yawAngle": -8.662049293518066}} LOG faces {} LOG faces {} LOG faces {"0": {"bounds": {"height": 314.3041666666667, "width": 159.25, "x": 227.5, "y": 457.3979166666667}, "leftEyeOpenProbability": 0.7734758257865906, "pitchAngle": 12.133540153503418, "rightEyeOpenProbability": 0.7252373695373535, "rollAngle": -81.5416488647461, "smilingProbability": 0.014574314467608929, "yawAngle": -10.743473052978516}} LOG faces {} LOG faces {} LOG faces {"0": {"bounds": {"height": 274.1375, "width": 136.5, "x": 223.5, "y": 427.775}, "leftEyeOpenProbability": 0.044218361377716064, "pitchAngle": 16.660924911499023, "rightEyeOpenProbability": 0.9922314882278442, "rollAngle": -56.60077667236328, "smilingProbability": 0.022218160331249237, "yawAngle": 2.201295852661133}} LOG faces {} LOG faces {} LOG faces {} LOG faces {} LOG faces {} LOG faces {} LOG faces {} LOG faces {} LOG faces {} LOG faces {"0": {"bounds": {"height": 262.0875, "width": 130.5, "x": 224.75, "y": 421.75}, "leftEyeOpenProbability": 0.7856113910675049, "pitchAngle": 15.452271461486816, "rightEyeOpenProbability": 0.9929311275482178, "rollAngle": -90.56649780273438, "smilingProbability": 0.017574554309248924, "yawAngle": -15.207443237304688}} LOG faces {} LOG faces {} LOG faces {} LOG faces {} LOG faces {"0": {"bounds": {"height": 267.10833333333335, "width": 137.5, "x": 200.75, "y": 417.73333333333335}, "leftEyeOpenProbability": 0.9445556402206421, "pitchAngle": 11.816859245300293, "rightEyeOpenProbability": 0.9922314882278442, "rollAngle": -90.85006713867188, "smilingProbability": 0.01241330523043871, "yawAngle": -19.47207260131836}}

(Note: These are just excerpts; the logs frequently show LOG faces {}, indicating no faces detected.)

Smartphone (please complete the following information):

Additional context The issue occurs in a custom project using the Expo bare workflow. The inconsistency in face detection makes it challenging to rely on this feature for any face-based operations within the app. Face detector was working seamlessly in face detector version 1.3.5.

iagoandrei commented 3 months ago

Please try react-native-vision-camera version: 4.0.0-beta.12. It's close to a stable release.

luicfrr commented 3 months ago

@divyaprakash0426 I've noticed on my test device (an old galaxy j5) that detections fail more often if my face is more close to camera, do this apply to you?

divyaprakash0426 commented 2 months ago

Please try react-native-vision-camera version: 4.0.0-beta.12. It's close to a stable release.

Tried, but getting same output with react-native-vision-camera: 4.0.0-beta.12

This was not the case with the earlier version: react-native-vision-camera-face-detector: 1.3.5

divyaprakash0426 commented 2 months ago

@divyaprakash0426 I've noticed on my test device (an old galaxy j5) that detections fail more often if my face is more close to camera, do this apply to you?

can't say with certainty, detections fail regardless.

luicfrr commented 2 months ago

I suppose this is related to wrong frame orientation... @divyaprakash0426 please try setting static values (try each 0, 90, 180 and 270) in replacement of rotation in bellow line and see if one of these values fixes this issue.

https://github.com/nonam4/react-native-vision-camera-face-detector/blob/5e249debc2ba4cb68b75f25456ade1a84f784131/android/src/main/java/com/visioncamerafacedetector/VisionCameraFaceDetectorPlugin.kt#L245

Can you also log rotation value? Is it different from the static value you defined?

Your code must be something like: 

val rotation = orientation!!.toDegrees()
Log.d(TAG, "rotation value: $rotation")
val image = InputImage.fromMediaImage(frameImage!!, 0) // also try 90, 180 and 270 values

Waiting for your feedback

luicfrr commented 2 months ago

closing due to lack of response