Closed Jordan-Terzian closed 11 months ago
That tutorial article's vision camera version is 2. Follow this page. You can still use that tutorial's codes partly, but that article have an issue that initialize pose detection instance every function call.
Added
Please refer to this code.
Hello, thank you for your help! Just wondering, how long until your library is available to use?
Unfortunately, Vision Camera 3.0.0 has an issue relative to Android (see this). So it is difficult to complete the integration right now. But the iOS side is almost complete, and if that issue is solved then it can be completed soon. The other option is to use version 2.
Hey, that tutorial is actually pretty good/nicely written. It just needs to be updated for VisionCamera V3 as I made some changes there to how Frame Processors work.
@ThatSongKim yep, I'll be working on fixing the imageFormat for Android this week! iOS already works perfectly fine btw.
I'm the creator of this tutorial on medium, and I just wanted to prepare updated version using VC v3, but unfortunately I have some kind of problem during that.
In native code everything works (but like @ThatSongKim said there is problem with initialising object on every call, so it's better to follow his lib code).
Anyway for presenting I use react-native-reanimated
to draw the pose, and the value should be updated from the frameProcessor's worklet type function.
const detectedPose = useSharedValue(defaultPose);
const frameProcessor = useFrameProcessor((frame) => {
'worklet';
// ...
detectedPose.value = pose;
});
But during update using this line: detectedPose.value = pose;
, I'm receiving this error:
Error: Reading from `_value` directly is only possible on the UI runtime, js engine: hermes
In new implementation of vision camera, there is necessary for use react-native-worklets-core
and I'm not sure that it's work correctly with "reanimated worklets". Based on current documentation it should work like before, but I think this is deprecated.
@mrousavy Do you have any tips for migration frame processors from VC v2 to v3? In your new tutorials / blog posts / examples I can't find any information about it.
Hey! Yea the Reanimated useSharedValue
does not work with Worklets. You need to import useSharedValue
from RN Worklets instead, and then it is also recommended to draw using RN Skia (useDrawCallback
) instead - as that is a powerful Canvas and doesn't use the UIKit view system, so it is more flexible and more powerful in drawing.
cc @wcandillon @chrfalch is useDrawCallback the right approach?
There is a PictureView API that can nicely be animated with Reanimated: https://github.com/Shopify/react-native-skia/blob/main/example/src/Examples/API/Icons/index.tsx#L77
Also the regular Canvas
element animates nicely with Reanimated.
Both allow for using the imperative API.
We also have an offscreen API might be useful there too. We don't recommend using useDrawCallback but of if there are use-cases that would make this API relevant, I would love to learn more about it.
Hey - again, this is part of the Skia integration that was not merged into main (yet). It is just very complicated. You can just render to another view (e.g. a Skia Canvas or a Reanimated View)
I'll close this for now, maybe in the future it will be integrated.
Question
Currently I'm trying to use googleML kit to make a pose detection screen. I've attempted to follow this tutorial:
https://medium.com/dogtronic/real-time-pose-detection-in-react-native-using-mlkit-e1819847c340
but all it's doing is processing frames, not highlighting the poses.
Would someone be able to please help me accomplish this? This is beyond my current skill level.
What I tried
I tried to follow @mrousavy blog (https://mrousavy.com/blog/VisionCamera-Pose-Detection-TFLite), however since skia implementation is no longer included, this doesn't appear to be working either.
My current implementation is identical to
https://medium.com/dogtronic/real-time-pose-detection-in-react-native-using-mlkit-e1819847c340
link to github: https://github.com/dogtronic/blog-pose-detection
but with the inclusion of react-native-worklets-core, and using the following package versions:
react-native: 0.72.3 react-native-reanimated: ^3.5.2 react-native-vision-camera: ^3.0.0 react-native-svg: ^13.11.0
VisionCamera Version
3.0.0
Additional information