Closed tcsantini closed 4 years ago
Christopher22
commented
4 years ago Amazing if this would work out, @tcsantini ! Just to mention it: Another version of PuRe for Pupil Labs is already out here. At the time of development, it was just not possible to include it due to the choosen Open Source License.
tcsantini
commented
4 years ago I've started decoupling the methods from EyeRecToo into a library at https://github.com/tcsantini/eyerec. PuRe and PuReST are already available.
The methods are accessible through Python with the following package: https://github.com/tcsantini/eyerec-python. This package is already pip-installable from the repository (although one has to locally clone it and update the submodule of the aforementioned library until I figure out how get pip to work with the submodule). I'll look into making the package available through PyPi.
papr
commented
4 years ago Dear @tcsantini,
Thank you for providing a nice overview of EyeRecToo features. At this point, I would like to clarify that the Pupil Core software ships with only one of the listed features [3], which is attributed in our documentation.
We have indeed implemented an open-source version of PuRe based on your paper [1]. An internal evaluation of the current 2D detector, the EyeRecToo PuRe implementation, and our own PuRe implementation has shown three things:
In other words, if we wanted to only detect 2d pupil ellipses we would replace the current 2D detector with the open-source PuRe implementation. However, in the default use case, the Pupil Core software uses the 2d pupil ellipses only as a stepping stone to build the 3d eye model. This requires only a few but very accurate ellipse estimations. Based on our evaluation, the current 2D detector is the better fit for this job due to its more precise confidence assignment. Independently of this evaluation, we are able to achieve higher frame rates with the current 2D detector. Therefore, we have decided to continue shipping the current 2D detector and not to replace it with our open-source PuRe implementation.
We are also not planning on shipping the PuReST pupil tracker by default.
Instead, we are working actively on making it easier for the community to integrate and provide core algorithms as user plugins. With Pupil Core v2.0, we have taken the first step in this direction by supporting user plugins implementing calibration procedures and gaze mapping algorithms. In the near future, we will support user plugins implementing pupil detection algorithms, too.
It would be great if someone would implement the core EyeRecToo algorithms as Pupil Core compatible plugins. If/when this happens, we would be happy to include the plugins in the Pupil community repository: https://github.com/pupil-labs/pupil-community
Therefore, we have decided to close this issue for now as we will not be implementing/adding EyeRecToo features to the default Pupil Core stack. We would be happy to continue the discussion via email or chat.
tcsantini
commented
4 years ago Dear @papr,
Thanks for the clarification!
Given the premise of easy situations, it makes sense to keep the 2D detector if it yields better discrimination of high quality estimates. In fact, PuRe's confidence does not represent probabilities, and I recommend just using a single threshold of 0.66. Would it be possible to share your evaluation with the community?
I think there was a bit of misunderstanding with the nomenclature here; my goal wasn't contributing method directly to Pupil Core. I'm glad to (slowly) add these methods as parts of the Pupil software stack (as Pupil plugins) -- tracking that was the purpose of this issue; I wasn't expecting you guys to maintain any of it :-)
The issue was created here because I was informed by email that your version of PuRe would find its way into Pupil Capture via an update around March. Thus, I thought it would be better to clarify and document what was already available from your side instead of guessing it based on the source code.
Anyway, a couple question regarding the pupil detection plugins:
papr
commented
4 years ago Hi @tcsantini
Please see my notes below regarding your technical questions:
PupilDetectorPlugin is the official entrypoint if you want to integrate a pupil detection algorithm into Pupil. The issue is that it is currently not yet possible to write these as user plugins (a). We are planning to support user pupil detector plugins within the next 2 releases.PupilDetectorPlugin should be powerful enough to support pupil tracking since the instance is free to keep its own state. If we are missing something particular that prevents pupil tracking please let us know.(a) User plugins are not shipped within the bundle but are required to be downloaded and placed in pupil_*_settings/plugins by the user.
[5] K. Dierkes, M. Kassner, A. Bullling, A novel approach to single camera, glint-free 3D eye model fitting including corneal refraction, ETRA ’18: Symposium on Eye Tracking Research and Applications, 2018 [6] K. Dierkes, M. Kassner, A. Bulling. A fast approach to refraction-aware eye-model fitting and gaze prediction. In ETRA ’19: Symposium on Eye Tracking Research and Applications, 2019
papr
commented
4 years ago With Pupil v2.6, it is now possible to create custom pupil detector plugins. We have also added a special section to our community repository to feature any community detectors.
This issue tracks the implementation of methods from EyeRecToo in the Pupil software stack. Foreseen are four methods:
1. The PuRe pupil detector [1]
Status: the Pupil team has already reimplemented this method in this repository.
2. The PuReST pupil tracker [2]
As far as I understand, Pupil doesn't differentiate pupil detectors (i.e., single frames) from pupil trackers (i.e., multiple frames). I guess this should be added as a pupil detector then?
Status: not started yet.
3. CalibMe calibration [3]
Status: The single marker calibration is already a CalibMe calibration (see https://github.com/pupil-labs/pupil-docs/pull/166). It however does not include any form of outlier removal, from which all calibration methods could benefit.
4. Grip gaze estimation [4]
Status: This should be mostly trivial to add. Pupil also makes use of the Świrski eye model and unprojections, so it should be a matter of making the unprojected ellipse's normal vector accessible to the gaze estimation layer. In EyeRec nomenclature, I refer to this as the instantaneous gaze vector (in contrast to the regular gaze vector yielded by the Świrski model). For the polynomial fit gaze estimation, it's just replacing the pupil coordinates by the normal vector angles in spherical coordinates. For the model-based gaze estimation, it is necessary to ignore the per-frame estimate eye center, replacing it by zeros, or by a fixed eye center estimated during calibration.
[1] Santini, Thiago, Wolfgang Fuhl, and Enkelejda Kasneci. "PuRe: Robust pupil detection for real-time pervasive eye tracking." Computer Vision and Image Understanding 170 (2018): 40-50.
[2] Santini, Thiago, Wolfgang Fuhl, and Enkelejda Kasneci. "PuReST: robust pupil tracking for real-time pervasive eye tracking." Proceedings of the 2018 ACM symposium on eye tracking research & applications. 2018.
[3] Santini, Thiago, Wolfgang Fuhl, and Enkelejda Kasneci. "Calibme: Fast and unsupervised eye tracker calibration for gaze-based pervasive human-computer interaction." Proceedings of the 2017 chi conference on human factors in computing systems. 2017.
[4] Santini, Thiago, Diederick C. Niehorster, and Enkelejda Kasneci. "Get a grip: slippage-robust and glint-free gaze estimation for real-time pervasive head-mounted eye tracking." Proceedings of the 11th ACM symposium on eye tracking research & applications. 2019.