search

asus4 / tf-lite-unity-sample

TensorFlow Lite Samples on Unity
870 stars 254 forks source link

MediaPipe - Iris Tracking Implementation #75

Closed FrankSpalteholz closed 3 years ago

FrankSpalteholz commented 4 years ago

Hey guys,

first of all you rock! I was waiting for so long having an Unity-package providing TF AND Mediapipe-features. I already tried out several examples on the iPad and except some of them everything is compiling like a charm so excellent job and thank you again very much for providing this. Let me ask you if there is either a way where i can get the iris tracking feature implementend else well or if that's not possible with the current package doin this myself are you guys planning to do so in near future? I'm working on a realtime-gaze-tracker where this could lead to much better results (at least i hope) rather than using old-school-opencv approaches. But again ... awesome job!

FrankSpalteholz commented 4 years ago

I was able to implement the iris tracking module myself on top of your face-mesh example. If you are interested I can share the code. Basically i was just adding the tf-module and an IrisDetect-class based on the FaceMesh-class. The only issue I'm facing is world-scale. The calculated eye keypoints are shrinked down a bit and I'm still trying to find out where this scale-offset compared to the face-mesh-keypoints is coming from.

asus4 commented 4 years ago

Thank you @FrankSpalteholz! Could you make a PR for the iris tracking?

FrankSpalteholz commented 4 years ago

Sure! Let me clean up my mess a bit and put in some notes and comments. It's also worthy to note that I stripped down the example project as much as possible so maybe pull it in an safe environment first. Again ... I'm super greatful for your rep!

FrankSpalteholz commented 3 years ago

Hey guys ... please check out this repo https://github.com/FrankSpalteholz/UnityMediapipeIris.git ... this is a stripped down version of your original one excluding all samples except the face-mesh scene, where i've added the iris-tracking. To be honest I don't know how much guidance you gonna need but let me shorty explain some of the small changes I did and what I've added:

  1. When you open the project you should open Assets->Scenes->FaceMesh
  2. By checking the hierarchy you gonna find a GO called "Detector" which provides a Detector-script that is controlling the whole thing.
  3. There is a string-field where you can type in your specific cam-name in case you have several cams connected to the computer (it SHOULD work with the default though)
  4. When you gonna study the Detector-script you will essentially find the same as in the original face-mesh-example except that i was allowing myself to create some simple "Init_xyz" + some "Draw_xyz" methods for cleaning up things a bit.
  5. I've commented out the face-mesh rendering but it works
  6. An issue that i couldn't figure out was, that under Windows for some reason the Face-Detection-output is working for the first loop and then it only detects 2 landmarks (the over-all detection works great so I haven't spend much time to investigate on this to be honest)
  7. The major work i did was implementing an extra class/script called "IrisDetect.cs" which was an adaption of the Facemesh-class. I'm pretty sure that there are redundant lines but to excuse myself, C# is not my main-programming-language so I did my best "adapting" your style and keep it simple. Feel free to optimize it and please let me know if there are any issues or questions. Hope it's going to work!

Edit: I've successfully tested it under iOS (with an iPadPro running 45fps), Windows and OSX

yugosaito4 commented 3 years ago

@FrankSpalteholz how did u get the iris tracking TF.lite file

FrankSpalteholz commented 3 years ago

From here https://github.com/google/mediapipe/tree/master/mediapipe/models ... "iris_landmark.tflite". Best F

yugosaito4 commented 3 years ago

@FrankSpalteholz thanks ! :) may i also ask, how did u seperate the eye area from the face mesh, as the eye area is yellow in color

stale[bot] commented 3 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.