💭 Running Custom Frame Processor Plugins on multiple threads

[aryamanshodhan]

Question

Hi! First of all, thank you so much for your continued work on this library. It is very well maintained and documented. I am working on a custom frame processor plugin containing a callback function (called detectObject) that runs a YoloV5 model on the native android end. This model provides its output every 2 seconds. I am running this function using runAsync, as shown in the code below:

`const frameProcessor = useFrameProcessor( frame => { 'worklet';

  runAsync(frame, () => {
    'worklet';
    const detection = detectObject(frame) as YoloDetection;
    updateYoloCoordinates(detection, frame.width, frame.height);
  });

[updateYoloCoordinates],

);`

That is, the coordinates for the yolo box is updating every 2 seconds. I was just wondering if you had suggestions for how I can do this faster. It would be good to be able to update the coordinates of the box every second vs. every 2 seconds. I tried creating multiple worklets running at different frame rates, but this did not seem to work.

`const frameProcessor = useFrameProcessor( frame => { 'worklet';

  runAsync(frame, () => {
    'worklet';
    const detection = detectObject(frame) as YoloDetection;
    updateYoloCoordinates(detection, frame.width, frame.height);
  });

  runAtTargetFps(0.5, () => {
    'worklet';
    runAsync(frame, () => {
      'worklet';
      const detection = detectObject(frame) as YoloDetection;
      updateYoloCoordinates(detection, frame.width, frame.height);
    });
  });

[updateYoloCoordinates],

);`

Reducing the size of the frame is difficult as well because the resolution is already 480 by 640, and reducing it further will make the video look quite pixelated. I am using a motorola g play (2021). Any suggestions and/or ideas (except making the YOLO model lighter) would be much appreciated. Please let me know if you require additional information. Thank you so much!

What I tried

As mentioned above, I tried creating multiple worklets running at different frame rates, but this did not seem to work.

VisionCamera Version

"react-native-vision-camera": "^4.3.2"

Additional information

• [ ] I am using Expo
• [X] I have read the Troubleshooting Guide
• [X] I agree to follow this project's Code of Conduct
• [X] I searched for similar questions in the issues page as well as in the discussions page and found none.

[maintenance-hans[bot]]

Guten Tag, Hans here.

[!NOTE] New features, bugfixes, updates and other improvements are all handled mostly by @mrousavy in his free time. To support @mrousavy, please consider 💖 sponsoring him on GitHub 💖. Sponsored issues will be prioritized.

[mrousavy]

Hey - thank you!

In this case, I would use native code to schedule async calls instead of runAsync. You could also try to use react-native-worklets-core to spawn multiple runtimes and balance between them, but at this point doing it in native is just easier