Mediapipe Solutions with the frame processor

[mks11]

Hi, I know react-native-fast-tflite can process any tflite model, but it looks like MediaPipe's pose estimation allows you to download models that is ".task" file (the underlying model are apparently tflite models) https://developers.google.com/mediapipe/solutions/vision/pose_landmarker

To work around this, I was looking into integrating the example Mediapipe android code into creating a frame processor plugin (followed steps from the react-native-camera-vision docs , but this doesn't seems as straightforward. (for one, I am not sure how to convert Frame into ImageProxy object, 2. I am mostly familiar with RN side of things)

Any help is appreciated.

[mrousavy]

Hey - this is unrelated to this library, but here's some pointers to help you anyways:

1. Frame.image contains the Image which you need to pass to MediaPipe.
2. MediaPipe can be integrated in FP Plugins, we've done that before it's not that hard. Building MediaPipe is a bit hard.

[computerjazz]

@mks11 did you have any luck getting MediaPipe up and running within react-native-vision-camera? I'm trying to do a similar thing with the Face Detection model, I'm getting resized camera images piped into the model, but the values I'm getting back aren't making much sense to me, and I'm making slow progress digging through the mediapipe source code to figure out why.

It'd be amazing if the general MediaPipe api could be more easily available from within React Native — and it sounds like that may be possible!

[mrousavy]

It'd be amazing if the general MediaPipe api could be more easily available from within React Native — and it sounds like that may be possible!

We're working on something like that internally at @margelo 👀