We need customization option of js version for mediapipe

[vkrytpo]

Hi guys, mediapipe is doing really great, and it's evolution rate is impressive.

problem: we can customize android, desktop, python verison of mediapipe but we can not do the same for js. we can build custom AAR, APK as per our requirement (which is much efficient), but we have no such things for js.

usecase: I have created a virtaul try-on for rings using mediapipe which is working fine, but i guess it need optimization for mobile, for that I need to add MedaiPipe PacketCloner as in model graph, but we can not do that normally.

tries: 1) I have also tried emscripten(to convert C++ to wasm, as you guys also using the same.) and tried to configure it but i guess it takes a lot to configure such a big project and I stucked at error https://github.com/google/mediapipe/issues/877. 2) Tried saving mediapie online graph runner https://viz.mediapipe.dev/demo/hand_tracking, it's working on my local (with custom graph-models) but i ended up a large js files( though I am trying to understanding it's structure). but as it's generic platform so it won't be that optimized. 3) Trying tensorflow library (ended up it's too slow, and almost no customization availble there.), and it is using same mediapipe official libraries at backend (also not synced with latest verison). 4) Trying https://cdn.jsdelivr.net/npm/@mediapipe/ it's working fine but in mobile fps are too low(5-6) which requires customization (addign a PacketCloner can fix this issue.), I am trying to customizing this.

final: either you guys could make it public (emscripten config for js(/third_party/mediapipe/web folder) - PackedSolution) or add PacketCloner as default with a switch to use it or not. or atleast share original hands.js(not uglified version)

A little help can spark something great.

[vkrytpo]

hey @sgowroji , @mhays-google let me know if I can get help from any of 4 options mentioned in problem.

[tyrmullen]

1. This is indeed a bigger project, and requires some more effort, so while it is something we'd like to do, it will be a little while before we can. It's not just a simple matter of exposing configs, because a lot of the sub-libraries themselves are different when being built outside of Google versus inside, so a decent amount of bazel/BUILD engineering will likely be necessary. However, I left a comment on that issue (#877) to hopefully help out any efforts there if that's an avenue people would like to pursue.
2. I'm surprised the js files are large-- there's a decent amount of JS code required to run the Wasm blob, but I'd imagine the majority of the size would be in the .wasm file, which (for the Visualizer) would just be much larger than you'd need, since it has to include lots of calculators. It should be relatively optimized too, but generally runs ML on CPU, so might not be as fast as the codepens (depending on your platform/device).
3. I can't really speak to this much at this time.
4. More customization here is something we're looking into as a logical next step. To kick things off, mhays@ has already released the Typescript defines, which should make the codebase more decipherable (see issue #1408). Hopefully that can help your current experiments while we figure out the customization roadmap!

[sgowroji]

Hi @vkrytpo, Did you get a chance to go through the above comment. Thanks!

[vKrypto]

@tyrmullen , thanks for the response.

I guess, I can try using ( issue #877) and (issue #1408)

[AndresPrada]

Hi @vKrypto, did you successfully deploy your own custom JS model following any of the mentioned approaches above? I'm much interested! Thanks!