search

Tobias-Fischer / rt_gene

RT-GENE: Real-Time Eye Gaze and Blink Estimation in Natural Environments
http://www.imperial.ac.uk/personal-robotics
Other
366 stars 68 forks source link

Implement face_encoding so that tracking maintains subject_id's are related to facial features #23

Closed ahmed-alhindawi closed 5 years ago

ahmed-alhindawi commented 5 years ago

Hello,

In a recent pull-request the issue of subject_id's came up with the use of UUID's initially but that broke downstream projects and thus reverted to counting.

The issue, aptly raised by @Twarz, is that if subject_1, subject_2 are being tracked, but then subject_1 is lost, the next time they appear they'll be labelled as subject_3.

IMHO, they should appear as subject_1 again.

One particular solution to this is to use a facial encoding - something similar to facial_encoding from dlib.

Would this be something that would be useful to people?

Tobias-Fischer commented 5 years ago

Hmm.. I guess yes, it would be useful, but it would also add additional code that needs to be maintained (and possibly yet another CNN that needs to fit in the GPUs memory). If it can be done in a few lines of code I think it's a great addition, but I'd be against it if it adds substantial complexity. Does that make sense?

ahmed-alhindawi commented 5 years ago

It does....There are three basic steps required for this it seems:

  1. Face detection
  2. Landmark extraction + alignment
  3. Facial encoding

Each of the above has footprints and there isn't a way around that as far as I can see...It is a compromise between memory and code overhead, the shortest code footprint would be to use dlib everything (i.e. face detector, landmark detector followed by facial encoding) but then we're definitely introducing more neural networks as the dlib solution isn't as good as the S3FD nor the FaceNet CNNs for the gaze estimation stuff.

We can definitely re-use the face detection and landmark extraction that we already do. All that's left to create is the alignment which is relatively small followed by running a CNN only one time when a new face gets added to the tracker in a manner similar to the assignment problem that @Twarz solved with the Hungarian optimiser....

I've finished the encoding part which I hope to incorporate into the tracker tomorrow; perhaps we can mull this over a pull request?

Alternatively, maybe we can provide hooks for the subject naming that we can customise externally to RT-GENE?

Tobias-Fischer commented 5 years ago

Hey, sure thing, I look forward to the pull request. Maybe it's a compromise to have an "opt-out" solution that can be disabled if not required. And if that opt-out solution is nicely separated from the core code, that would make me happy :)

Tobias-Fischer commented 5 years ago

Fixed in #24. Thanks a lot @ngageorange