luicfrr / react-native-vision-camera-face-detector

Vision Camera Frame Processor Plugin to detect faces using MLKit Face Detector
https://www.npmjs.com/package/react-native-vision-camera-face-detector
MIT License
75 stars 16 forks source link

How can I still operate on the Frame? #28

Closed frodriguez-hu closed 2 months ago

frodriguez-hu commented 3 months ago

The plugin looks awesome btw.

Describe the bug Hey, I am trying to use this plugin, and after I receive the data and wehn I want to access the original frame, I am able to do it, but the issue is that I am not able to execute any other frame processor plugin cause the Frame was marked as Closed. Any idea on how to solve this?

I am getting this error when I am trying to resize it:

Error: Exception in HostFunction: com.mrousavy.camera.core.FrameInvalidError: [capture/frame-invalid] Trying to access an already closed Frame! Are you trying to access the Image data outside of a Frame Processor's lifetime?

To Reproduce Steps to reproduce the behavior:

  1. Go to '...'
  2. Click on '....'
  3. Scroll down to '....'
  4. See error

This is the frame processor:

    detectFaces(frame, handleFacesDetection, {
      classificationMode: 'all',
      landmarkMode: 'all',
      performanceMode: 'accurate',
      returnOriginal: true,
    });

And the callback:

if (Object.keys(result.faces).length <= 0 || !result.frame.original) {
        return;
      }
      console.log('asd1');
      const {bounds} = result.faces[0];
      const {width, height, x, y} = bounds;
      currentBounds.current = bounds;
      try {
        const resized = resize(result.frame.original, {
          scale: {
            width: 112,
            height: 112,
          },
          crop: {
            height,
            width,
            x,
            y,
          },
          rotation: '0deg',
          pixelFormat: 'rgb',
          dataType: 'uint8',
        });
        console.log(resized);
      } catch (e) {
        console.log(e);
      }

Expected behavior A clear and concise description of what you expected to happen.

Logs and Screenshots If applicable, add logs and/or screenshots to help explain your problem.

Smartphone (please complete the following information):

Additional context Add any other context about the problem here.

luicfrr commented 3 months ago

I suppose you’re using my recommended way, am I right? If this the case you need to use default vision camera docs way then

luicfrr commented 3 months ago

Or you can call some frame internal method. I don’t remember right now what exactly but it’s something like increseRef. But you will need to cast frame as internalFrame type

felire commented 3 months ago

I resolved this by disabling the originalFrame prop, and returning the result on the detectFaces instead of executing the callback. By doing that I was able to use rhe result data on the frame for other operations, this worked fine on android I haven't tested it on ios. In my opinion I thini returning the result instead of the callback would be a better spproach for the plugin, what is the advantage that you have by using the callback?

luicfrr commented 2 months ago

@felire can you provide an example code of your sollution? So other people with same issue can test it too.

About "returning result instead..." I'd appreciate it if you could open a PR with your ideas 😁

frodriguez-hu commented 2 months ago

I just did this @nonam4 :

export function detectFaces(
  frame: Frame,
  callback: CallbackType,
  options?: FaceDetectionOptions
) {
  'worklet'
  if ( !plugin ) {
    throw new Error( 'Failed to load Frame Processor Plugin "detectFaces"!' )
  }
  // @ts-ignore
  const result: DetectionResult = plugin.call( frame, options )
  callback( result )
  return result;
}

Returning the result and using it instead of using the callback