Face api is incompatible with new versions of tensorflow.js 2.x, 3.x

[katmatus]

face-api.js requires 1.x version and it fails with newer versions like 2.x, 3.x and gives type errors because of the compatibility issues

[vladmandic]

faceapi has not been updated for a long time, so that's main reason why this port exists:
https://github.com/vladmandic/face-api

[jeffreytgilbert]

Recent posts on TensorFlow's blog make me think maybe there is another model that could be loaded up and used for fast results and more detailed info.

https://blog.tensorflow.org/2020/11/iris-landmark-tracking-in-browser-with-MediaPipe-and-TensorFlowJS.html

[jeffreytgilbert]

Oh, also, this is happening https://www.chromestatus.com/feature/5678216012365824 And this https://blog.tensorflow.org/2020/07/tensorflow-2-meets-object-detection-api.html And most importantly, this happened https://blog.tensorflow.org/2020/04/tensorflow-lite-core-ml-delegate-faster-inference-iphones-ipads.html which will eventually manifest as this https://webmachinelearning.github.io/webnn/ and then we shouldn't have to worry about performance anymore.

[vladmandic]

@jeffreytgilbert

Recent posts on TensorFlow's blog make me think maybe there is another model that could be loaded up and used for fast results and more detailed info.

I've been working with those models (and few others) for a while now, check out: https://github.com/vladmandic/human

Oh, also, this is happening https://www.chromestatus.com/feature/5678216012365824

Is great if all you need is to be able to authenticate user - that's what it's for, nothing else.

And this https://blog.tensorflow.org/2020/07/tensorflow-2-meets-object-detection-api.html

Not sure how that relates? It's a great high-level API for learning, but any SOTA object detection model doesn't rely on OD API.

And most importantly, this happened https://blog.tensorflow.org/2020/04/tensorflow-lite-core-ml-delegate-faster-inference-iphones-ipads.html

A way to natively use Apple's proprietary NPU - great performance speedup on Apple's new silicone.

which will eventually manifest as this https://webmachinelearning.github.io/webnn/

One more GPU/TPU/NPU accelerated backend? Just because it's committee driven, doesn't make it better than existing ones. E.g, tfjs-node-gpu + cuda already do that job if you have nVidia GPU.

I'm far more interested in development and adoption of WebGPU (to be used as a next-gen WebGL).

[jeffreytgilbert]

WebGPU is great, but that shouldn't be the only way Apple and everyone access those NPUs. My focus has been on the performance improvements using efficient non-blocking APIs. Face/object detection in the browser APIs is not feature rich, however with web standards, they start small and get better from there. That's why I'm following that spec and WebNN/OpenVino/NNAPI/etc. They aren't drop in replacements for anything here, but might be used for simple tasks in a non-blocking way.

You might be thinking, but currently this code isn't blocking! Fair, but 2 years ago when i was using it i hit a GPU bottleneck that blocked main ui rendering because the integrated graphics chip couldn't do both and it's stuck with me ever since.

[vladmandic]

Agree, WebGPU use-case is in-browser, not catch-all.

But regarding WebNN, etc. - I don't see how a committee driven backend designed for high-level usage can accomplish much other than increase entry-level adoption of ML (great for quick learning). All really good models require a lot more low-level ops and that is not the goal of WebNN. E.g., I don't care about pre-packaged "person detection", I want to be able to use best available model.

Also, most money in ML research is coming from corp sponsors that develop their own backends (e.g., FB has PyTorch, Google has Tensorflow, Alibaba has MNN, Apple has CoreML, Tenecent has NCNN, nVidia has TensorRT, etc.) - and I need to be able to run models developed in their native frameworks. Notice that Apache foundation and OpenAI are pushing for MXNet adoption, but it really hasn't picked up so far.

Now, if I can easily convert those models to ONNX (as "universal" model format) and then run them using WebNN - great. But I see that as always being couple of steps behind cutting edge as new ops need to be implemented downstream.

[jeffreytgilbert]

You're more versed in this than I am. My last touchpoint was 2019. That's not how I understood WebNN's approach as documented in their workflow hierarchy graphics and spec documentation, but I'm fully ok with being way off in my interpretation.

Here, it's mentioned what the roles are: https://webmachinelearning.github.io/webnn/#programming-model

So my understanding is that WebNN would be a common interface for loading, queuing and executing work against low level system APIs available to the native code in the browser and that the common interface would not look to be a replacement for WebGPU, WebGL, etc, but instead a standard way to access them and get async results back from work that may block a thread. There would still be a javascript framework (Tensorflow for example) which decided which models to load and which of the available APIs to choose, similar to how Canvas allows you to choose a 2D or 3D context for instance, but the dev/framework is still responsible for implementing compatible instructions.

I'm not deep enough into it to have any insight into what you mentioned regarding apache and ONNX, but that sounds like it might be cool. I wonder if Kronos or OASIS standards workgroups will pick something like that up so it ends up working like Vulcan does in replacing OpenGL with a universal API for common low level GPU tasks. I'll have to read up on it more when I'm done with my backlog of other reading materials. ;)

[jeffreytgilbert]

PS, happy you're involved and posting on these tickets! Can't make magic happen without magicians!

[vladmandic]

I definitely don't see a scenario where TFJS loads a model and then passes it to WebNN for execution - why would devs maintain TFJS then? Just as a loader? TFJS would not run on top of WebNN - both of them are frameworks that implement low level functions using some CPU/GPU/TPU/NPU acceleration. And in case of browser, how do you get to underlying hardware? Via WebGL or (in the future) WebGPU (which then internally use something like OpenGL or Vulcan or D3D - that is up to browser implementation).

But...WebNN cannot have a parser and loader for every model format that comes from different framework, they have to pick one. I hope they pick ONNX as that is supposed to be "universal" format.

I can see how someone uses Tensorflow to develop a model, then converts it to ONNX and runs it using WebNN - so role of Tensorflow to develop models stays, but role of TFJS to run models at the edge diminishes if (and that's a big if) WebNN picks up. But given it's several steps from original model, I see it as always being several steps behind.

Still, PyTorch (for example) doesn't have browser-native solution. Actually, none of the other frameworks do, only TF does in form of TFJS (that's the reason why I choose to start with TF/TFJS when I started with ML 6 months ago). So WebNN may pick up given the need to run all those other models in browser.

Anyhow, I get the goals of WebNN, but if underlying interface to HW is flaky as WebGL is, they can talk about non-blocking as much as they want, but they need better way to interface with HW. And if WebGPU solves that (and uses Vulcan in the backend), it will solve it for TFJS as well.

[withinJoel]

faceapi has not been updated for a long time, so that's main reason why this port exists: https://github.com/vladmandic/face-api

Thank you so much mate, Without you I would not be able to fix my issue for my project. Thank you very much mate. And I also would be very happy if you can review my project and tell me your opinion on my project.