Error loading hands-mediapipe in a Chrome extension

[dajuric]

I wanted to use hands mediapipe (Javascript) in a Chrome extension but I have encountered an error, where the library will not load at all.

Chrome extension where the error below happens is located here: chrome-extension-DEBUG.zip

Error loading data:

[sgowroji]

Hi @dajuric, During load unpack there is no screen or camera pop up (default popup). Could you explain more about your code changes and use case. Thanks

[dajuric]

HI @sgowroji. I have not changed anything. If you leave everything as it is, the mediapipe files are loaded from the web (local files are provided and loaded only for the second case where files are loaded locally), and the sample which is executed as a content script is also taken as it is. You should be able to see the camera activity indicator if you go to www.google.com and if you open the developers tools there you will see the error.

Let me know if I can help you more.

Best.

[dajuric]

Any progress ?

[mhays-google]

To get this working in an extension, you will need to package all of the files from NPM into your extensions directory and then change the loader path so that the '.data' and '.wasm' (and any other data files) are read from that same location. I currently worked with a few people doing a hackathon where they made this work in an extension, so I'm certain its doable. I have not made an extension myself, but I'm happy to promote it once I've got a solid example.

[dajuric]

HI @mhays-google.

I have put all the files in a local directory called Scripts. The extension with the unmodified files in a local directory is available in my first post. chrome-extension-DEBUG.zip. That generates the error which screenshot is in the first post.

Next, I have modified hands_solution_packed_assets_loader.js (line 26) where I hard-coded the path to a requested asset. That cleared the XmlHttpRequest error, but the following error still remains. I would love to tackle this issue, but the source of the hands.js is minified and obfuscated so I can not do that.

Can I somehow give you more info or something else so that issue can be resolved ?

[dajuric]

Any update on this ?

[dennis-gojitech]

I would be great to get an update. I am facing the same issues as @dajuric (see image). The error happens after a hands.send(). I've gotten similar code running fine as a standalone webapp. It would be helpful to see the hackathon code, if possible.

[mhays-google]

We don't recommend hand editing the minified loader to embed a path. The issue is still that you are not plugging in the correct path to locateFile when creating a new instance of Hands.

The hackathon code is internal to Google, I won't be able to release it.

My advice is to monitor the fetches and see the path that is described in the 404. If it is not the same path in which you are storing files, you'll need to modify locateFile. And though you've probably solved this, you may also need to configure your web server to properly serve those files.

[dennis-gojitech]

From what I can tell, after hacking the minified code, all the files (js, data, binarypb) are loaded properly (painful but it works). I will try to set the correct path with locateFile to see if that changes anything. I am now investigating using tfjs directly instead of mediapipe-hands. Can you tell us if the hackathon code used mediapipe-hands directly or other libraries, like tfjs?

[tyrmullen]

I believe the hackathon code mentioned was using the MediaPipe JS Hands Solution directly. Also, for locateFile usage, perhaps the comments on this issue can be of help: https://github.com/google/mediapipe/issues/1812

[dajuric]

@dennis-gojitech Have you successfully embedded the library into the extension ? (so that it works from the extension) If so, can you tell me which files need to be modified ? Thanks.

[google-ml-butler[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you.

[google-ml-butler[bot]]

Closing as stale. Please reopen if you'd like to work on this further.

[google-ml-butler[bot]]

Are you satisfied with the resolution of your issue? Yes No

[hongzimao]

@dennis-gojitech did you get it to fully work with an extension? I hacked the loader js to load all files properly as well, but I'm still hitting the error in your screenshot (on Sept 15) whenever I use hands.send(). Thanks!