optical flow tracking of the facelandmarks in video/live webcam feed

[abhishek-peri]

Please make sure that this is a feature request. As per our GitHub Policy, we only address code/doc bugs, performance issues, feature requests and build/installation issues on GitHub. tag:feature_template

System information

• TensorFlow.js version (you are using):
• Are you willing to contribute it (Yes/No):

Describe the feature and the current behavior/state. wanted to check if anyone tried to implement optical flow (LK) for tracking in between frames instead of predicting the landmarks on each and every frame when trying on live webcam or videos.

Will this change the current api? How?

Who will benefit with this feature?

Any Other info.

[gbaned]

cc @pyu10055

[vladmandic]

@abhishek-peri i've played with it a bit, but it's too much math for javascript - execution time ends up slower than model inference.

btw, current blazeface + facemesh combo already skips detection if boxes are known and just modifies boxes coordinates depending on mesh results, so it skips detection inference as much as possible and just runs mesh inference (although that algorithm is a bit buggy, took me a while to fix it)

[Honga1]

I've had some success adding acceleration and damping on the key points, essentially doing extrapolation between face-mesh frames to reduce jitteriness, and make the response more fluid. Wouldn't be so difficult if you took the blazeface bounding box coords, which update more often, and then scaled the accelerations accordingly.

If the end I some multi order low pass filters to the movement, as it seemed less jitter was more satisfying than faster response times.

[google-ml-butler[bot]]

Closing as stale. Please @mention us if this needs more attention.

[aoxipo]

Hello, Have you tried opencv.js ? it provided Optical flow method here