mediapipe conversion

[lghasemzadeh]

Hello,

I need to use these libraries in my python scripts but they are written in C++ and provided for JS. https://blog.tensorflow.org/2020/11/iris-landmark-tracking-in-browser-with-MediaPipe-and-TensorFlowJS.html https://google.github.io/mediapipe/solutions/face_mesh https://github.com/vladmandic/human

Can I convert them to python using your converter? I actually looking for a model/library/package for facemesh and iris detection to use them in my model. I am writing an algorithm in python and I don't know JavaScript and C++.

[patlevin]

Hi,

the converter can only help with converting the model data to Tensorflow (Python). The code you need to work with the models (preparation of inputs and evaluating outputs) cannot be converted automatically.

I did a python version of the old facemesh version, but didn't publish it because it was of limited use (to me anyway). Unfortunately I didn't keep it, so I'm afraid I cannot help you with that.

If it's not too urgent, I might be able to create a python package for the new Face Landmarks Detection model, which includes Iris detection.

[lghasemzadeh]

@patlevin thank you very much for your response.

I REALLY need the converted of the new version of facemesh and iris detection, It is not very urgent. I am new to these topics but I can help you if there is anything that I can do :)

Did you also checked the third link ı sent you?

[patlevin]

Did you also checked the third link ı sent you?

I did. I am familiar with Vladimir's work and he actually contributed to this tool, too by giving very valuable feedback. I am pretty impressed by what he managed to build 👍

[lghasemzadeh]

actually he suggested your repo!

Do you have any estimation about the time you can open that converted version repo?

[patlevin]

Do you have any estimation about the time you can open that converted version repo?

It's difficult to make a realistic estimation. The models contain quite a lot of setup and post-processing code as well as a bunch of browser-specific stuff. I will first have to think about a reasonable way to turn this into a Python library (input/output types, etc.).

I'll have a closer look at it over the weekend and maybe get a first draft done by Sunday or so. But no promises 😉

[lghasemzadeh]

ok, thank you so much. Actually I don't care about browser-specific stuff. I just need new version of mediapipe facemesh and iris detection to use the waypoint/coordinates as input for my algorithm. Then please let me know whenever you did :)

[patlevin]

@lghasemzadeh It's been a while and I didn't forget about your request. Unfortunately what I did forget about was publishing the repo. You can find my try at converting the MediaPipe code to Python here: Face Detection for Python

I included a little example for iris recolouring. I'll put it on PyPi later as well.

[lghasemzadeh]

@patlevin Thank you very much Patrick. I am curious to see the outputs as soon as I get rid of errors.

Just a question: To find the iris (iris detection), did you used a pre-trained model as used here? As far as I understood you used image processing techniques such as ROI, masking and contouring to find iris location, right?

Actually, I was doing sth similar: face detection --> tensorflow or dlib face landmarks --> eye region landmarks --> ROI, masking and contouring to find iris. BUT at the end I got very low accuracy for realtime video streams. That is why I decided to use MediaPipe and this package because for the iris detection part they use Deep Learning. I mean they collected data, labeled iris and train a model, not image processing and maths. that is why these packages seems more robust in finding iris.

Waiting for your response

thx

[patlevin]

@lghasemzadeh

To find the iris (iris detection), did you used a pre-trained model as used here?

Yep, I used the exact same model. Accuracy isn't great and depends on the image quality and (funnily enough) the eye colour. It works great with high contrast eyes (e.g. blue or grey eyes and well-lit environments) and not so great with input from webcams.

I described the details here and the code with all the details can be found here.

There are a couple of ways to improve this, but they would increase latency a lot. One approach included gamma-correction and noise reduction on the input image before running the detection pipeline. Performing detection and recolouring on gamma-corrected input works much better.

Hope that helps

[lghasemzadeh]

Thank you very much for your help and guide :)