What is the difference between mediapipe on the web vs tensorflow js?

delebash commented 4 years ago

For example the mediapipe web version of MediaPipe Hands https://google.github.io/mediapipe/solutions/hands.html has Multihand support. However the tfjs version https://github.com/tensorflow/tfjs-models/tree/master/handpose does not support this feature. Is there a simple example of using the MediaPipe webassembly version just like the tensorflow version?

delebash commented 4 years ago

Hi, did you by chance find an answer? The problem I am having is that wasm has a much lower fps. Example run the tensorflow js handpose demo at https://storage.googleapis.com/tfjs-models/demos/handpose/index.html. If you change the backend from webgl to wasm the fps drops dramatically. I would like to use some of the newer applications in mediapipe such as iris tracking but mediapipe uses wasm and tensoflow js does not have a version. I thought wasm was supposed to be faster than javascript.

Thanks for your help, Dan

yousefamar commented 4 years ago

This is not an answer to your original quesion, but regarding wasm vs WebGL vs JS.

WebGL allows you to use hardware acceleration, which translates to being able to being to perform hundreds of parallel calculations, perfect for many ML use cases. These calculations run on your GPU, not your CPU. Wasm is much lower-level than vanilla JS, meaning less granular overhead, and I believe also allows multi-threading out of the box, so you can take advantage of all your CPU cores for speed. Still it will be nowhere near as fast when it comes to parallel computing. Vanilla JS (without web workers anyway) is single-threaded and has more overhead. So it's not surprising that the WebGL backend is faster for this. Most devices will support hardware-accelerated WebGL, but not all.

rahulscp commented 4 years ago

I found the mediapipe version are much better than the tfjs models bt sadly i haven't figured out yet how to use that in a web application. I saw the visualizer where it run these models in browser as wasm. Is there any way to use https://google.github.io/mediapipe/solutions/hands.html In a web application like how i can use the tfjs version ?
Searched everywhere for a sample unfortunately never found any

annxingyuan commented 4 years ago

Certain MediaPipe models are available through the TF.js model repository, such as face-landmarks-detection and handpose. TF.js models include a JavaScript API, and can be used with the TF.js WebAssembly or WebGL accelerated backends.

Although TF.js aims to replicate the behavior of MediaPipe models, they are independent codebases and some drift is unavoidable. The postprocessing pipelines in particular may evolve separately. Both teams are actively considering options to address this issue.

@delebash - for what it's worth, we just added iris detection to our face-landmarks-detection model via MediaPipe Iris.

delebash commented 4 years ago

Thank you for the information and its great to hear Iris tracking has been added, awesome. Do you know if the tfjs models are as accurate in detection as the mediapipe models? I have been doing some lite testing and it appears that the medipipe version seems to be more accurate than the tfjs version. I haven't compared numbers but it just looks that way. Is there any documentation or can you confirm the difference between mediapipe C++, mediapipe web assembly, tfjs WebGel accuracy. I have read an article on the framerate differences but it doesn't tell me the accuracy difference in point detection.

Thanks, Dan

vasanthhr commented 4 years ago

@annxingyuan

Thanks for your updates.

Do you know when you can improve tfjs-handpose frame-rate and add hand detection feature as like mediapipe-handpose ?

yuccai commented 4 years ago

Let me know if I understand correctly :

tfjs-models API implements algorithms in javascript with tensorflow-js, it must load models with tfjs format and can run algorithms on different backends (CPU, WebGL, WebGPU, WebAssembly)
MediaPipe visualizer demos use the WebAssembly port of MediaPipe, a javascript program will coordinates what WebAssembly functions to call, for instance process to process a frame on CPU, processGL to process frame on GPU. It means that computationnal functions such as model loading or inferencing are executed with WebAssembly. That allow us to load .tflite models.

I noticed performances and quality differences between theses 2 demos (like @delebash ) : tfjs-demo and mediapipe-demo.

The tfjs demo is less accurate and runs with lower FPS than the MediaPipe one.

Where does it come from ?

[ ] Are the model different (different architecture, different dataset, different training) ?
[ ] The model are the same but the format differences (tfjs vs tflite) brings differences?
[ ] Does it mean that tensorflow-js is less performant than MediaPipe WebAssembly port ?

jrouet commented 3 years ago

@yuccai MediaPipe is really more powerfull than TF. You can compare a virtual background on https://meet.google.com, and virtual background implemented on https://beeweet.com.