search

tensorflow / models

Models and examples built with TensorFlow
Other
77.02k stars 45.78k forks source link

@tensorflow-models/handpose running too slow on React Native #9652

Open romildojuliano opened 3 years ago

romildojuliano commented 3 years ago

Hi everyone. I'm trying to develop an app to process sign language using React Native, Expo and @mediapipe/handpose, but the performance I get is too slow for what we need. The smartphone I'm testing with is the Samsung Galaxy S9, which is not at all a bad phone, and the FPS I get is around 2 ~ 6 fps; The process is very simple: I load the handpose model -> Then with the CameraWithTensors from TensorFlow Js React Native I can create a loop to process the frames from the smartphone camera -> Inside this loop I'm able to call the model.estimateHands();

But when the estimatehands method is called, even not rendering the camera, the performance drops absurdly, making it almost useless for what we want because we need to apply another model to process the points generated; from the examples I've seen at the mediapipe website, the results I'm getting are very uncongruent. Do you think the problem is being caused by my code? Here's a video on how it's looking like:

https://user-images.githubusercontent.com/33132257/104114231-c2d7b300-52e0-11eb-8f5f-294029ede81d.mp4

Here's some sample code: 20210110_013138.jpg 20210110_013402.jpg 20210110_013304.jpg

(Sending photos because, as you can see at the edit history I couldn't manage to format it properly 🤭)

ravikyram commented 3 years ago

@romildojuliano

Please, fill issue template.. Provide the exact sequence of commands / steps that you executed before running into the problem. Thanks!

romildojuliano commented 3 years ago

You can check our repository: https://github.com/romildojuliano/Silent-Spell

uffoltzl commented 3 years ago

@romildojuliano did you find a way to optimize your application?

techTutorialsYTube commented 3 years ago

Any way to optimize the model for react native

gmundewadi commented 2 years ago

Set a PREDICTION_INTERVAL = 60. This would ensure that you only load a tensor into memory every 60th frame. See how its done in the following link:

https://talmpro.com/2020/04/18/build-react-native-app-with-tensorflow-js-and-mobilenet-part-2/

coucoseth commented 1 year ago

@gmundewadi seems the link you provided is dead, Did anyone find a solutoin to this?