💭 Frame processing FrameInvalidError seems to be thrown arbitrarily

[aryamanshodhan]

Question

Hey! This library has been extremely helpful for my app, so I really appreciate your continued effort in maintaining and developing this library. I am using frame processors in the following way in my code:

const frameProcessor = useFrameProcessor(
    frame => {
      'worklet';

      runAsync(frame, () => {
        'worklet';
        const detection = detectObject(frame) as YoloDetection;
        updateYoloCoordinates(detection, frame.width, frame.height);
      });

      const OCR_FPS = 0.5;
      runAtTargetFps(OCR_FPS, () => {
        'worklet';
        const data = scanText(frame);
        const ocrText = data.resultText;
        updateOcrText(ocrText);
      });
    },
    [updateOcrText, updateYoloCoordinates],
  );

Here are the functions updateOcrText and updateYoloCoordinates:

const updateYoloCoordinates = useRunOnJS(
    (detection: YoloDetection, frameWidth: number, frameHeight: number) => {
      const scaledDetection = detection
        ? scaleCoordinates(detection, frameWidth, frameHeight)
        : undefined;
      setYoloCoordinates(scaledDetection);
    },
    [],
  );

const updateOcrText = useRunOnJS((newText: string) => {
    setOcrText(prevState => ({
      ...prevState,
      text: newText,
    }));
  }, []);

Just for come context, scanText is the frame processor plugin provided by the community plugin react-native-vision-camera-text-recognition. detectObject is a custom frame plugin that I have implemented which uses a yolo model. It takes about 2 seconds to execute and output coordinates of a bounding box, which are scaled using the scaleCoordinates function. Even though I am running the yolo model on the frame asynchronously, every so often I get this error:

I understand what the error means, but I am not sure why it occurs. Any insight for how to fix this would be much appreciated! Thank you so much!

What I tried

I am not sure how to fix this issue. There is no obvious pattern I observe for when the error pops up. It seems to be coming up on arbitrary frames. I tried using frame.incrementRefCount() before each of the functions running on the JS thread, but typescript complains saying that Property 'incrementRefCount' does not exist on type 'Frame'..

VisionCamera Version

"react-native-vision-camera": "^4.3.2"

Additional information

• [ ] I am using Expo
• [X] I have read the Troubleshooting Guide
• [X] I agree to follow this project's Code of Conduct
• [X] I searched for similar questions in the issues page as well as in the discussions page and found none.

[maintenance-hans[bot]]

Guten Tag, Hans here.

Thank you for your detailed issue report! It's good to hear ze library is helpful for you. However, for the error you are experiencing, it seems like you might need to ensure that all frame processing and ref counting are done correctly.

Please check if you're properly managing the lifecycle of your frames. Also, remember that you need to use incrementRefCount() correctly corresponding to the frame's lifecycle.

If you want more immediate support from @mrousavy or ze team, consider sponsoring ze project. We appreciate your understanding!

Note: If you think I made a mistake by closing this issue, please ping @mrousavy to take a look.

[vishalyad16]

@aryamanshodhan How can I reduce the size of a video that is 151 MB for a 1-minute recording? The bitRate setting does not seem to be working. https://github.com/mrousavy/react-native-vision-camera/issues/3113

[aryamanshodhan]

Thank you for the response! However, the error message seems to be saying that I'm trying to access the width property of the frame beyond its life cycle. The only place I am accessing frame.width is in the runAsync call. Shouldn't runAsync allow me to use frame properties for as long as it is executing all the logic inside it?

[mrousavy]

hm yea this should work, can you still try to use increment and decrement ref count nevertheless? ignore the TS errors, it's just so users don't use this.

[aryamanshodhan]

Thanks so much for your quick response! Just want to clarify how to modify the code, since I did not find a documentation for incrementRefCount and decrementRefCount.

const frameProcessor = useFrameProcessor(
    frame => {
      'worklet';

      runAsync(frame, () => {
        'worklet';
        frame.incrementRefCount();
        const detection = detectObject(frame) as YoloDetection;
        updateYoloCoordinates(detection, frame.width, frame.height);
        frame.decrementRefCount();
      });

      const OCR_FPS = 0.5;
      runAtTargetFps(OCR_FPS, () => {
        'worklet';
        frame.incrementRefCount();
        const data = scanText(frame);
        const ocrText = data.resultText;
        updateOcrText(ocrText);
        frame.decrementRefCount();
      });
    },
    [updateOcrText, updateYoloCoordinates],
  );

[mrousavy]

No the idea is to increment it when you're thread-hopping.

This is the same that runAsync already does, so incrementRefCount() before runAsync, then decrementRefCount() at the end inside of runAsync.

Also you created a question, not a bug report, so frankly I cannot really help you as the template is not filled out (eg no logs).