victordibia / handtrack.js

A library for prototyping realtime hand detection (bounding box), directly in the browser.
https://victordibia.com/handtrack.js/
MIT License
2.83k stars 250 forks source link

model question: relationship to main repo, input sizes, unknown kernel ops #61

Open vladmandic opened 2 years ago

vladmandic commented 2 years ago

i just stumbled on your model and so far i really like it

i do have few questions:

what is the relationship of two different models?

notes state that tfjs graph model is just converted from tf model, but model signatures,
internal operations and sizes (when manually converted and equally quantized) do not match?

what is the correct model input size?

model input size is marked as dynamic shape: [1, -1, -1, 3],
but models model names indicate 320 for ssd320fpnlite and 640 for ssd640fpnlite ?
and notes further confuse the issue since they state that both models are trained on inputs 450 * 380 ???

tfjs graph model has unknown operations

tensorlistreserve, enter, tensorlistfromtensor, merge, loopcond, switch,
exit, tensorliststack, nextiteration, tensorlistsetitem, tensorlistgetitem,
reciprocal, shape, split, where

those ops are not supported by tfjs, but it seems it doesn't impact model execution at all
(defined but unused?)

and one request for enhancement - predict hand rotation

my use case is so the hand detection model can be comined with other models
(e.g. detailed finger tracking, gesture analysis, sign language recognition, etc.)

the thing is, all those models are trained on vertially oriented hand
if hand detection model returned approximate rotation angle, then image could be
rotated and cropped before being used for further analysis