How to make a pose detection screen ❓

[Jordan-Terzian]

Question

Currently I'm trying to use googleML kit to make a pose detection screen. I've attempted to follow this tutorial:

https://medium.com/dogtronic/real-time-pose-detection-in-react-native-using-mlkit-e1819847c340

but all it's doing is processing frames, not highlighting the poses.

Would someone be able to please help me accomplish this? This is beyond my current skill level.

What I tried

I tried to follow @mrousavy blog (https://mrousavy.com/blog/VisionCamera-Pose-Detection-TFLite), however since skia implementation is no longer included, this doesn't appear to be working either.

My current implementation is identical to

https://medium.com/dogtronic/real-time-pose-detection-in-react-native-using-mlkit-e1819847c340

link to github: https://github.com/dogtronic/blog-pose-detection

but with the inclusion of react-native-worklets-core, and using the following package versions:

react-native: 0.72.3 react-native-reanimated: ^3.5.2 react-native-vision-camera: ^3.0.0 react-native-svg: ^13.11.0

VisionCamera Version

3.0.0

Additional information

• [ ] I am using Expo
• [X] I have read the Troubleshooting Guide
• [X] I agree to follow this project's Code of Conduct
• [X] I searched for similar questions in the issues page as well as in the discussions page and found none.

[thatsongkim]

That tutorial article's vision camera version is 2. Follow this page. You can still use that tutorial's codes partly, but that article have an issue that initialize pose detection instance every function call.

---

Added

Please refer to this code.

[Jordan-Terzian]

That tutorial article's vision camera version is 2. Follow this page. You can still use that tutorial's codes partly, but that article have an issue that initialize pose detection instance every function call.

Added

Please refer to this code.

Hello, thank you for your help! Just wondering, how long until your library is available to use?

[thatsongkim]

Unfortunately, Vision Camera 3.0.0 has an issue relative to Android (see this). So it is difficult to complete the integration right now. But the iOS side is almost complete, and if that issue is solved then it can be completed soon. The other option is to use version 2.

[mrousavy]

Hey, that tutorial is actually pretty good/nicely written. It just needs to be updated for VisionCamera V3 as I made some changes there to how Frame Processors work.

@ThatSongKim yep, I'll be working on fixing the imageFormat for Android this week! iOS already works perfectly fine btw.

[lukaszkurantdev]

I'm the creator of this tutorial on medium, and I just wanted to prepare updated version using VC v3, but unfortunately I have some kind of problem during that.

In native code everything works (but like @ThatSongKim said there is problem with initialising object on every call, so it's better to follow his lib code).

Anyway for presenting I use react-native-reanimated to draw the pose, and the value should be updated from the frameProcessor's worklet type function.

const detectedPose = useSharedValue(defaultPose);

const frameProcessor = useFrameProcessor((frame) => {
      'worklet';
      // ...
      detectedPose.value = pose;
});

But during update using this line: detectedPose.value = pose;, I'm receiving this error:

Error: Reading from `_value` directly is only possible on the UI runtime, js engine: hermes

In new implementation of vision camera, there is necessary for use react-native-worklets-core and I'm not sure that it's work correctly with "reanimated worklets". Based on current documentation it should work like before, but I think this is deprecated.

@mrousavy Do you have any tips for migration frame processors from VC v2 to v3? In your new tutorials / blog posts / examples I can't find any information about it.

[mrousavy]

Hey! Yea the Reanimated useSharedValue does not work with Worklets. You need to import useSharedValue from RN Worklets instead, and then it is also recommended to draw using RN Skia (useDrawCallback) instead - as that is a powerful Canvas and doesn't use the UIKit view system, so it is more flexible and more powerful in drawing.

cc @wcandillon @chrfalch is useDrawCallback the right approach?

[wcandillon]

There is a PictureView API that can nicely be animated with Reanimated: https://github.com/Shopify/react-native-skia/blob/main/example/src/Examples/API/Icons/index.tsx#L77 Also the regular Canvas element animates nicely with Reanimated. Both allow for using the imperative API.

We also have an offscreen API might be useful there too. We don't recommend using useDrawCallback but of if there are use-cases that would make this API relevant, I would love to learn more about it.

[mrousavy]

Hey - again, this is part of the Skia integration that was not merged into main (yet). It is just very complicated. You can just render to another view (e.g. a Skia Canvas or a Reanimated View)

I'll close this for now, maybe in the future it will be integrated.