Laser Eye : Gaze Estimation via Deep Neural Networks

Installation

Requirements

• Python 3.5+
• Linux, Windows or macOS
• mxnet (>=1.4)

While not required, for optimal performance(especially for the detector) it is highly recommended to run the code using a CUDA enabled GPU.

Run

• Prepare an usb camera or some videos
• [Optional] More Accurate Face Alignment
  • Download Hourglass2(d=3)-CAB pre-trained model
  • Replace MobileAlignmentorModel with CabAlignmentorModel
• python3 video_gaze_test.py

Tips

• EditMxnetDetectionModel's scale parameter to make a suitable input size of face detector
• More details at Wiki

Gaze Estimation for MMD Face Capture

• Try with Open Vtuber

Face Detection

• RetinaFace: Single-stage Dense Face Localisation in the Wild
• faster-mobile-retinaface (MobileNet Backbone)

Facial Landmarks Detection

• MobileNet-v2 version (1.4MB, using by default)
• Hourglass2(d=3)-CAB version (37MB)

Head Pose Estimation

• head-pose-estimation

Iris Segmentation

• U-Net version (model size 71KB, TVCG 2019)

Citation

@article{wang2019realtime,
  title={Realtime and Accurate 3D Eye Gaze Capture with DCNN-based Iris and Pupil Segmentation},
  author={Wang, Zhiyong and Chai, Jinxiang and Xia, Shihong},
  journal={IEEE transactions on visualization and computer graphics},
  year={2019},
  publisher={IEEE}
}

@inproceedings{deng2019retinaface,
  title={RetinaFace: Single-stage Dense Face Localisation in the Wild},
  author={Deng, Jiankang and Guo, Jia and Yuxiang, Zhou and Jinke Yu and Irene Kotsia and Zafeiriou, Stefanos},
  booktitle={arxiv},
  year={2019}
}