Facemesh not working locally

[geneticmoo]

Facemesh doesn't work locally when including the ml5.js library. Posenet works OK.

The code to call up face predictions is

  facemesh = ml5.facemesh(video, modelReady);
  facemesh.on("predict", getPredictions);

In the Chrome browser the following error message is reported when not connected to the internet :

ml5.js:1778 GET https://tfhub.dev/tensorflow/tfjs-model/blazeface/1/default/1/model.json?tfjs-format=file net::ERR_INTERNET_DISCONNECTED

t.fetch @ ml5.js:1778
(anonymous) @ ml5.js:401
(anonymous) @ ml5.js:401
(anonymous) @ ml5.js:401
(anonymous) @ ml5.js:401
n @ ml5.js:401
t.load @ ml5.js:401
(anonymous) @ ml5.js:2718
(anonymous) @ ml5.js:2718
(anonymous) @ ml5.js:2718
(anonymous) @ ml5.js:2718
__awaiter @ ml5.js:2718
e.load @ ml5.js:2718
(anonymous) @ ml5.js:2718
(anonymous) @ ml5.js:2718
(anonymous) @ ml5.js:2718
(anonymous) @ ml5.js:2718
__awaiter @ ml5.js:2718
loadGraphModel @ ml5.js:2718
load @ ml5.js:97293
loadDetectorModel @ ml5.js:97293
load$1 @ ml5.js:97293
_callee$ @ ml5.js:97414
tryCatch @ ml5.js:114391
invoke @ ml5.js:114625
prototype.<computed> @ ml5.js:114443
step @ ml5.js:2491
(anonymous) @ ml5.js:2509
F @ ml5.js:2795
(anonymous) @ ml5.js:2488
loadModel @ ml5.js:97448
Facemesh @ ml5.js:97393
facemesh @ ml5.js:97553
setup @ sketch.js:65
_setup @ p5.js:62985
_start @ p5.js:62908
p5 @ p5.js:63261
_globalInit @ p5.js:62189
Promise.then (async)
280.../core/main @ p5.js:62214
o @ p5.js:34
(anonymous) @ p5.js:38
267../accessibility/color_namer @ p5.js:54831
o @ p5.js:34
r @ p5.js:51
(anonymous) @ p5.js:55
(anonymous) @ p5.js:18
(anonymous) @ p5.js:20

There is also a second error message which looks the same but instead of trying to get the file

https://tfhub.dev/tensorflow/tfjs-model/blazeface/1/default/1/model.json?tfjs-format=file

it is looking for

https://tfhub.dev/mediapipe/tfjs-model/facemesh/1/default/1/model.json?tfjs-format=file

To fix this I though maybe I could just download these json models and include them in my local folder alongside the ml5.js file. But then in the ml5.js code I have no idea how to point the code at these local files.

Please has anyone solved this and got Facemesh running completely locally?

Thanks

[tlsaeger]

Hi @geneticmoo thanks for opening this issue. I had the very same a few months ago. There is project which made a lot of the models available offline: https://github.com/ml5js/ml5-data-and-models-server but it was not maintained in the past. Facemesh and Handpose where quite recently added that's why they are not included.

I think it would be cool to work on the offline functionality, but at the moment this is not supported in ml5.js. If you dig deeper into Tensorflow.js, you could make it work. I ended up using a cellular hotspot in my project, maybe this could work for you as well?

Cheers, Tom

[geneticmoo]

I'm developing pose estimation for 1) exhibitions and 2) overseas online courses.

The exhibitions may be long (say 3 months) and the local internet may be slow / intermittent and I would be happier if all the models were local. I will try and understand how the posenet and bodypix models work locally on the link you've suggested

For the online courses I'd like it if all the models required could be included with the test code. The course is in China, where access to some online javascript/json/google libraries are restricted.

Thanks

[tlsaeger]

Gotcha!

As I mentioned this is not supported at the moment, but if you find out a workaround please share it here.

Also thanks for the issue in the other repo. (ml5js/ml5-data-and-models-server#12)

[joeyklee]

Hi @geneticmoo! - Thanks for starting this discussion here! Hi @tlsaeger! - thanks for your replies and tips!

I haven't been able to work on ml5 in a long time, but this is a good reminder that it might be time to think about how to update to some of the more recent tensorflow versions (latest is 3.14 we are currently running 1.2.9 😬 ).

Facemesh is no longer a package that is being maintained and has been archived -- https://www.npmjs.com/package/@tensorflow-models/facemesh#this-repository-has-been-archived-in-favor-of-tfjs-modelsface-landmarks-detection-and-will-no-longer-be-updated -- in favor of the face-landmarks-detection model -- https://www.npmjs.com/package/@tensorflow-models/face-landmarks-detection.

If we do end up updating our tf versions to some of the more recent versions, then it looks like in the latest face-landmarks-detection lib we can specify where our model files should be loaded from -- https://github.com/tensorflow/tfjs-models/tree/master/face-landmarks-detection/src/mediapipe#create-a-detector -- which, in this case, would be somewhere on a local server.

[sproutleaf]

Moved the issue to https://github.com/ml5js/ml5-next-gen/issues/16