🐛 : V3, useSharedValue is not working with V3

rbayuokt commented 11 months ago

What's happening?

hi Marc, I'm currently working on developing a vision-camera-pose-detection project with MLKit. I initially wrote the native code for iOS, and it connected perfectly. However, I encountered a problem when attempting to draw the overlay for the skeleton based on the keypoints.

In general, I used useSharedValue from Reanimated V3 to pass a value. when I use useSharedValue I receive an error message like this: Reading from '_value' directly is only possible on the UI runtime.

I have already wrapped my JS function into a worklet, but I still get an error.

If I pass the value into setState it works, but it doesn't behave as expected. there is a delay, as shown in this video. this is my friend who tested it

https://github.com/mrousavy/react-native-vision-camera/assets/49748249/1517267b-fcd5-4125-9715-97a561803644

"react-native-vision-camera": "^3.0.0", "react-native-worklets-core": "^0.2.0", "react-native-reanimated": "^3.4.2"

Reproduceable Code

PoseDetection.ts

import {VisionCameraProxy, Frame} from 'react-native-vision-camera';
import {IPoseDetection} from '../utils/frameProcessor/PoseDetection';

const plugin = VisionCameraProxy.getFrameProcessorPlugin('poseDetection');

export function poseDetection(frame: Frame): IPoseDetection {
  'worklet';

  if (plugin == null) {
    throw new Error('Failed to load Frame Processor Plugin "poseDetection"!');
  }

  return plugin.call(frame) as unknown as IPoseDetection;
}

Main Code:

...
// POSE

  const pose = useSharedValue<IPoseDetection>(defaultPose);

  const convertToWorklet = (poseObject: IPoseDetection, frame: Frame) => {
    const xFactor = dimensions.width / frame.width;
    const yFactor = dimensions.height / frame.height;

    const poseCopy: IPoseDetection = {
      leftShoulder: {x: 0, y: 0},
      rightShoulder: {x: 0, y: 0},
      leftElbow: {x: 0, y: 0},
      rightElbow: {x: 0, y: 0},
      leftWrist: {x: 0, y: 0},
      rightWrist: {x: 0, y: 0},
      leftHip: {x: 0, y: 0},
      rightHip: {x: 0, y: 0},
      leftKnee: {x: 0, y: 0},
      rightKnee: {x: 0, y: 0},
      leftAnkle: {x: 0, y: 0},
      rightAnkle: {x: 0, y: 0},
    };

    Object.keys(poseObject).forEach(v => {
      const key = v as keyof IPoseDetection;
      poseCopy[key] = {
        x: poseObject[key].x * xFactor,
        y: poseObject[key].y * yFactor,
      };
    });

    pose.value = poseCopy;
  };

  const setToWorklet = Worklets.createRunInJsFn(convertToWorklet);

  const frameProcessor = useFrameProcessor(
    frame => {
      'worklet';
      const poseObject = poseDetection(frame);
      console.log('pose data', poseObject);
      setToWorklet(poseObject, frame);
    },
    [pose],
  );

Relevant log output

Error: Reading from `_value` directly is only possible on the UI runtime, js engine: hermes

Camera Device

No response

Device

iPhone 11 ( 15.5 )

VisionCamera Version

3.0.0

Can you reproduce this issue in the VisionCamera Example app?

[ ] I can reproduce the issue in the VisionCamera Example app.

Additional information

[ ] I am using Expo
[ ] I have enabled Frame Processors (react-native-worklets-core)
[X] I have read the Troubleshooting Guide
[X] I agree to follow this project's Code of Conduct
[X] I searched for similar issues in this repository and found none.

emclain commented 11 months ago

I think I've been able to fix this by using useSharedValue from react-native-worklets-core instead of reanimated.

Per https://github.com/mrousavy/react-native-vision-camera/pull/1466,

Reanimated is no longer used as a Worklet Runtime. Instead, VisionCamera now uses react-native-worklets-core.

emclain commented 11 months ago

However, this presents a problem if I want to use the shared state in an animation. Reanimated doesn't recognize when changes to the state occur, so they don't trigger animation update.

mrousavy commented 11 months ago

Yea useSharedValue is from RN Worklets Core, not from Reanimated.

rbayuokt commented 11 months ago

Yea useSharedValue is from RN Worklets Core, not from Reanimated.

mmm, it still doesn't work when I pass it into useAnimatedStyle. The line is not rendered. I was using react-native-svg and converted the Line into an AnimatedLine with react-native-reanimated's 'createAnimatedComponent'. I'm sorry, but is there any way to draw an overlay like an animated line?

rbayuokt commented 11 months ago

the native modules for Android and iOS is perfectly connected, it shows the values. I'm just missing the last step for drawing the overlay 😢 after it's finish I will publish this vision-camera frame processor library

cardanoeconews commented 11 months ago

I also have the same issue. I've successfully created my own version of vision-camera-ocr that does text recognition. Using latest react-native-vision-camera (v3).

The missing piece is how to get data from the worker in JS, out to my react component, and rendered.

When using "Worklets.createRunInJsFn" the app (iOS) crashes, when calling my backToReactThread function ^ It crashes in Native code, inside jsi::String JSCRuntime::createStringFromUtf8 calling: JSStringCreateWithUTF8CString

And using useShareValue (from Worklets) seems to have no affect. useAnimatedStyle ignores it..

How can we get some of the results/data from our useFrameProcessor javascript function, to the rest of react native so we can draw ui updates?

mrousavy commented 11 months ago

Hey! So VisionCamera V3 uses react-native-worklets-core, and react-native-reanimated uses it's own Worklets implementation, so the useSharedValue is not compatible with the one from Reanimated (I think we could add interop support for that though, wdyt @tomekzaw?)

What you can do (and what I would suggest you to do anyways regardless of this issue) is to draw the entire overlay using @Shopify/react-native-skia. This is much faster and flexible/powerful for drawing such overlays. Use the draw callback to draw, that should also work with useSharedValue. (afaik, right @chrfalch?)

cardanoeconews commented 11 months ago

Thank you @mrousavy ! Btw, is there a type-o in the offiical docs where it says:

"(Optional) If you want to use Frame Processors, you need to install react-native-worklets-core 1.0.0 or higher."

I think react-native-worklets-core only goes up to version 0.2.0.. I don't see 1.0.0 ?

Separately, I am still seeing the crash inside jsi::String JSCRuntime::createStringFromUtf8 calling: JSStringCreateWithUTF8CString . Do you have any tips or hints on where I can look for that? I wonder if it's because I have 0.2.0 and not 1.0.0 of react-native-worklets-core (which is no longer available on npm)?

cardanoeconews commented 11 months ago

@mrousavy I wonder if you could share a package.json from a working setup. Curious to see what version of react-native, react-native-worklets-core, and react-native-vision-camera are all working well together.

mrousavy commented 11 months ago

Package.json; the one here in example/ app.

And yea, you're right the docs should say 0.2.0 instead of 1.0.0.

I need a reproduceable example for that

cardanoeconews commented 11 months ago

@mrousavy thank you. that example was super helpful. I managed to get it building and running. In the end I had to:

Upgrade Node !! This was the biggest source of my problems. I was on Node v16... Needed to go to 18+..
I needed to install string-hash-64, I know you're aware since I saw you post in the other git repo about it
I needed to apply the _closure --> __closure (double underline) patch from the other thread to prevent a crash on startup with the example app.

After that, I was good to go. Thank you so much for your help.

mrousavy commented 10 months ago

Oh interesting! Could you maybe send a PR to react-native-worklets-core to fix the _closure issue? Thanks! cc @chrfalch

I'll close this for now, let me know if this issue is still active and we can re-open it

tomekzaw commented 10 months ago

just fyi, in https://github.com/software-mansion/react-native-reanimated/pull/4803 we renamed _closure to __closure for the purpose of consistency so that all fields start with a double underscore.

mrousavy commented 10 months ago

Ah shoot haha okay yea we'll rename as well then 😅

mrousavy commented 10 months ago

Thanks for the heads up Tomasz!

umbertoghio commented 8 months ago

Hi, if somebody else stumbles on thid issue my (ugly) solution is to copy the useSharedValue variable from the main thread:

import { useSharedValue, Worklets } from 'react-native-worklets-core'
import { useSharedValue as useSharedValueR } from 'react-native-reanimated'

  const pose = useSharedValue(getDefaultPose())
  const poseR = useSharedValueR(getDefaultPose())

  const updateSkeleton = Worklets.createRunInJsFn((p) => {
    poseR.value = { ...p }
  })

const frameProcessor = useFrameProcessor(frame => {
    'worklet'
...
 updateSkeleton(pose.value)

mrousavy commented 8 months ago

Will add compat for that soon!

c-info commented 8 months ago

Hi, if somebody else stumbles on thid issue my (ugly) solution is to copy the useSharedValue variable from the main thread:

import { useSharedValue, Worklets } from 'react-native-worklets-core'
import { useSharedValue as useSharedValueR } from 'react-native-reanimated'

  const pose = useSharedValue(getDefaultPose())
  const poseR = useSharedValueR(getDefaultPose())

  const updateSkeleton = Worklets.createRunInJsFn((p) => {
    poseR.value = { ...p }
  })

const frameProcessor = useFrameProcessor(frame => {
    'worklet'
...
 updateSkeleton(pose.value)

I tried above thing but seems not working UI is not updating or showing

umbertoghio commented 8 months ago

Verify first that updating your reanimated shared value from js works , then put a console log in the Worklets.createRunInJsF to verify it has been called

I tried above thing but seems not working UI is not updating or showing

c-info commented 8 months ago

Verify first that updating your reanimated shared value from js works, then put a console log in the Worklets.createRunInJsF to verify it has been called

I tried above thing but seems not working UI is not updating or showing

ys function has been calling but the UI is not updating unless a hard refresh means I have to update the state value to show the block. I'm using the latest version of vision-camera

jslok commented 8 months ago

Same, it is not working for me. @umbertoghio what version vc, wc, and reanimated are you using?

edit: nevermind, got it to work. Thanks for the fix!

c-info commented 8 months ago

@jslok may you send some snippets to make it workable? and please mention relevant versions for reanimated and vision camera and worklet-core

jslok commented 8 months ago

@jslok may you send some snippets to make it workable? and please mention relevant versions for reanimated and vision camera and worklet-core

vc 3.6.16 worklet-core 0.24 reaniated 3.6.1

Could not save an object in useSharedValue so I had to split it up into four different ones.

const rectWidth = useSharedValue(100) // rect width
  const rectHeight = useSharedValue(100) // rect height
  const rectX = useSharedValue(100) // rect x position
  const rectY = useSharedValue(100) // rect y position

  const rectWidthR = useSharedValueR(100) // rect width
  const rectHeightR = useSharedValueR(100) // rect height
  const rectXR = useSharedValueR(100) // rect x position
  const rectYR = useSharedValueR(100) // rect y position

  const updateRect = Worklets.createRunInJsFn((frame) => {
    rectWidthR.value = frame.width
    rectHeightR.value = frame.height
    rectXR.value = frame.x
    rectYR.value = frame.y
  })

  const frameProcessor = useFrameProcessor((frame) => {
    'worklet'
    const rectangle = cardDetect(frame)

    if (rectangle.x) {
      const { width: frameWidth, height: frameHeight } = frame

      const xFactor = windowWidth / frameWidth
      const yFactor = windowHeight / frameHeight

      //return
      // Assign animated values to rect shared values
      rectWidth.value = rectangle.width * xFactor
      rectHeight.value = rectangle.height * yFactor
      rectX.value = rectangle.x * xFactor
      rectY.value = rectangle.y * yFactor

      updateRect({
        width: rectWidth.value,
        height: rectHeight.value,
        x: rectX.value,
        y: rectY.value,
      })
    }
  }, [])

  // Animated styles for the box
  const animatedStyle = useAnimatedStyle(() => {
    return {
      width: withSpring(rectWidthR.value),
      height: withSpring(rectHeightR.value),
      transform: [
        { translateX: withSpring(rectXR.value) },
        { translateY: withSpring(rectYR.value) },
      ],
    }
  })

mrousavy / react-native-vision-camera