What is the recommended way to draw landmarks over the video❓

[adrian-bobev]

Question

What is the recommended way to draw position landmarks of body parts over the live video?

What I tried

I tried to create a simple MLKit Pose detection FrameProcessor. At the moment I am successfully logged the detected pose landmarks in the console. However, I am wondering what is the recommended way to draw those landmarks.

I saw some mentions of Skia, but not sure what is the current state and how I should use it.

VisionCamera Version

3.6.13

Additional information

• [ ] I am using Expo
• [X] I have read the Troubleshooting Guide
• [X] I agree to follow this project's Code of Conduct
• [X] I searched for similar questions in the issues page as well as in the discussions page and found none.

[rodgomesc]

i’m using react-native-skia with stable 30fps

[adrian-bobev]

But how you do it? I saw that blog post: https://mrousavy.com/blog/VisionCamera-Pose-Detection-TFLite

However, the code is using ^3.0.0-rc.7 and from that page: https://react-native-vision-camera.com/docs/guides/skia-frame-processors I see that it is removed from the official release.

This is the reason why I am searching now for the recommended way how this can be implemented

[rodgomesc]

ik this is a bit confusing but vision-camera don't have integration with skia natively anymore, this is on hold for now, so you just collect those points you need and use rn skia shapes to draw them on a screen https://shopify.github.io/react-native-skia/docs/shapes/ellipses, it will require some math to normalize between the screen rendered image and the point coordinates but notting to much complex, you can ask for help on discord magelo comunity as well

[adrian-bobev]

Thabjs for the answer. Just for future reference what I have done is the following:

export default function App() {
  const [position, setPosition] = useState();
  const device = useCameraDevice('front');

  const onObjecteDetected = Worklets.createRunInJsFn(pos => {
    setPosition(pos);
  });

  const frameProcessor = useFrameProcessor(frame => {
    'worklet';

    const obj = detectObject(frame);

    onObjecteDetected({
      x: obj?.x,
      y: obj?.y
    });
  }, []);

  if (!device) {
    return null;
  }
  return (
    <>
      <Camera
        frameProcessor={frameProcessor}
        style={StyleSheet.absoluteFill}
        video={true}
        device={device}
        isActive={true}
        videoHdr={false}
      />
      <Canvas style={StyleSheet.absoluteFill}>
        <Circle cx={position.x} cy={position.y} r={10} color="red" />
      </Canvas>
    </>
  );
}

I am just using createRunInJsFn in order to update the state of the app component and to rerender it on every frame and use the position properties to draw.

[rodgomesc]

try to stay far from the javascript thread, use useValue from skia instead of useState

[adrian-bobev]

@rodgomesc, is there example of that because I cannot really understand how it should be done. If I do not use useState, how the App component will know to update itself on each frame to rerender the circle from Skia?

PS: I tried useSharedValue from react-native-worklets-core and useValue from react-native-skia, but it seems neither of both works. The App component is not update and the circle or react-native-skia is not moved at all.

[adrian-bobev]

I manage to make it work. However, the useValue works only if the the value stored in it is number, I tried to use object { x: 10, y: 20 }

for example and it did not work.

PS: However, I just realized that this is not exactly I wont, because the skia overlay is not recorded in the video since it is not part of the frame. I guess in order to do it, I need to do the landmarks drawing in the native frame processor, am I right?

[mrousavy]

Yea, recording that will be quite tricky.

The native Skia implementation would've handled all of that, but as @rodgomesc said, I removed it from the repo because it got way too complex for all the people that were not using Skia.

[adrian-bobev]

@mrousavy, was there an issue with the implementation with native skia implemnetation or it was just because of the complexity that this change will bring to the library?

In other words, is it safe to find the change and use it or there was unresolved issue which was the cause of revert?

[mrousavy]

The issue was just the complexity of the codebase because I had two codepaths, one Skia and one native.

It's somewhere in the V3 PR, you'll find it :)

[adrian-bobev]

Thanks for the answers. WIth that I think we can close the issue since.

[mrousavy]

See https://github.com/mrousavy/react-native-vision-camera/pull/2727 👀

[adrian-bobev]

Sounds promising!