search

victordibia / handtrack.js

A library for prototyping realtime hand detection (bounding box), directly in the browser.
https://victordibia.com/handtrack.js/
MIT License
2.83k stars 250 forks source link

Many false detections while running on mobile device #47

Open dajuric opened 3 years ago

dajuric commented 3 years ago

The library runs fine on my laptop. However, when I run the online demo (even on provided images) on my mobile device (Samsung Galaxy S10e) I get a lot of false positives which confidences are 0.999.

Do you know what is happening ?

victordibia commented 3 years ago

Hi @dajuric ,

I am still trying to figure this out myself. On my IphoneX, detection on images work correctly in safari and chrome. Video struggles a bit (possibly due to low FPS) . I have not tested on Android, but would expect images to work corrrectly.

The one thing that comes to mind is that handtrack.js is designed to use a webgl backend (webgl should be available in most modern mobile/web browsers). When that is not available, I think it kicks all computation to cpu. That in itself should not result in a wild change in accuracy. There is also the issue with numeric precision on mobile webgl backends that can impact computations.

On mobile devices, WebGL might only support 16 bit floating point textures. However, most machine learning models are trained with 32 bit floating point weights and activations. This can cause precision issues when porting a model for a mobile device as 16 bit floating numbers can only represent numbers in the range [0.000000059605, 65504]. This means that you should be careful that weights and activations in your model do not exceed this range. To check whether the device supports 32 bit textures, check the value of tf.ENV.getBool('WEBGL_RENDER_FLOAT32_CAPABLE'), if this is false then the device only supports 16 bit floating point textures. You can use tf.ENV.getBool('WEBGL_RENDER_FLOAT32_ENABLED') to check if TensorFlow.js is currently using 32 bit textures.

What browser are you using on your Galaxy device? Perhaps we can check its supported numeric precision? Also, the new version of handtrack.js seen in the demo (still ongoing work) provides a toggle for fp16 (medium) and int8 (small) quantized versions of the model. Does the mobile accuracy issue persist across all model sizes?

-V.

dajuric commented 3 years ago

Hi Victor.

thanks for the response. I am using Google Chrome (mobile) - version 89.

Regarding model sizes - I can not find the toggle in the referenced demo page. Once enabled, I will gladly test it.

P.S. just for the reference, I wanted to mention the Hand-JS (I am the coo-creator) - the library was built a while ago. It has been built on older technology (not CNN) - it did not succeed unfortunately, but again I was not pushing hard due to other reasons. I can see that yours is totally different story. And I wish you good luck.