search

zaidalyafeai / ml-projects

Implementation of web friendly ML models using TensorFlow.js. pix2pix, face segmentation, fast style transfer and many more ...
MIT License
647 stars 254 forks source link

gaze tracking with poser #4

Open Sohalt opened 5 years ago

Sohalt commented 5 years ago

Do you think it would be possible to expand the poser case to do rudimentary gaze tracking?

zaidalyafeai commented 5 years ago

I need more details about the task.

Sohalt commented 5 years ago

What I was thinking about was roughly guessing the position on the screen a person is looking at from the webcam image, similar to the service offered by https://xlabsgaze.com. My specific use case is, that I would like to automatically focus the window I am looking at. The accuracy for this does not need to be very high (10-20cm should be enough), but I do not want to involve additional cameras, other than the webcam already set up.

zaidalyafeai commented 5 years ago

It will be difficult because you might have your head still but look at different places in the screen.

Sohalt commented 5 years ago

While the head might stay still, the pupil will look in different directions however. So what I was thinking was, using the poser network to locate the eyes, then feed a high resolution part of the image cropped to the area around the eyes into a network, together with the overall position and direction of the head, as determined by the poser network and try to correlate it with a position on the screen. To train the network one would capture video of the person in front of the camera while asking them to follow a moving dot on screen with their eyes.

zaidalyafeai commented 5 years ago

Seems like a great idea.