Eyetracking

[alexboche]

Is your feature request related to a problem? Please describe. Mouse movement is extremely useful. Eyetracking provides this capability for people who cannot use a mouse. Current features like text manipulation don't cover every situation, especially long range movements. Grids are much worse than eyetracking. Without eyetracking, you have to make a bunch of commands separately for every app you use.

Describe the solution you'd like The best eyetracking paradigm I've seen, after trying a few, is to have the eyes control big movements and the head make little adjustments. @Versatilus 's system is nice and works only with eyes (not head) but the big movements are a bit too slow since eyes also control little movements. talon implements the combined eye+head tracking system with just the 4c hardware. I suggest emulating this.

Some other desirable features:

• ability to deactivate temporarily by closing one eye (as in versatilus' system)
• special edge cases near the border of the screen
• possible retroactive clicking so it clicks where the mouse was a couple miliseconds ago so you don't have to "click to where the puck is going"

Describe alternatives you've considered A clear and concise description of any alternative solutions or features you've considered.

I know there was some discussion of waiting till the next gen hardware comes out, which should be soon I guess? But I think the 4c is plenty good enough, so no need to wait as I see it.

Additional context Key people with knowledge of this stuff: @Versatilus , @shervinemami (if he can see this) has a lot of expertise in software-hardware interaction, computer vision and I believe eye tracking too; ryan might also be willing to share some insights as well. The developers of precision gaze mouse and project iris might be other people to contact for more info if necessary.

[lexxish]

I'd think creating python bindings for https://developer.tobii.com/consumer-eye-trackers/stream-engine/ in Caster would be the place to start since this is the API recommended by Tobii support for non Tobii pro users, however it seems like EyeXMouse may use a different API so perhaps @Versatilus will know why that is and whether we should bind to the same API EyeXMouse uses instead.

I'd also think head tracking with Tobii 4c and TrackIR would be helpful along with mouse zoom like Talon https://www.youtube.com/watch?v=VMNsU7rrjRI

Some relevant links: Eye tracker in C: https://github.com/Versatilus/EyeXMouse Eye tracker and head tracker in C#: https://github.com/PrecisionGazeMouse Recommended API by Tobii unless you pay for Tobii Pro (thousands of dollars): https://developer.tobii.com/consumer-eye-trackers/stream-engine/

[LexiconCode]

tobii getstarted

Reviewing Select the hardware with associated supported Eye Tracking Core Software. Most embedded eye tracking was updated in 2018 to v2.13.4

Whereas tobii C4 IS4 may have been updated in 2019 as its version is v2.16.5

Confirmed that IS3 platform is end of life. Stated reason for price increase for US customers.

Hardware that supports Tobii Aware maybe considered the NexGen hardware IS5.

[LexiconCode]

Wolfmanstout has made a lot of progress in integrating eye tracking as a part of his grammars. Perhaps we could pursue permissions to reuse some of his work to implement eye tracking in caster.

https://github.com/wolfmanstout/dragonfly-commands/blob/master/_eye_tracker_utils.py

[ileben]

I have integrated eye tracking with 4C into my grammars. I also have optical character recognition so that i can capture screen image next to current gaze location and parse the text in the image, to click on the spoken word. This allows me to click on any text on the screen with maximum precision. It is all here: https://github.com/ileben/EyeTrackingHooks My tools are implemented in c#, with a C++/CLI wrapper which i invoke from python via ctypes. Hope you find any of this useful.

[altV]

hi! is there a way to get tobii 4c working for linux? if yes, do you know if there are python bindings?

[LexiconCode]

hi! is there a way to get tobii 4c working for linux? if yes, do you know if there are python bindings?

This is something I would love to see.The overall issue and why I can't do directly integrated into caster without the user doing some steps to set up is to avoid violating their license. This has to do with distribution and the type of APIs used some of which require research license.

I have plans to integrate OCR and Eye Tracking libraries see the link for demo and explanation. Once a PR is merged into dragonfly dealing with Windows DPI scaling this will be integrated into Caster.

It may be possible use stream engine and load the appropriate dlls/api through Python. If that is the case it will be very similar to what's implemented for Windows. From there we can work on Linux support to its underlying libraries gaze-ocr and screen-ocr

There is so much the the Caster project any help would be appreciated as per your interest. Keep in touch if I can help facilitate or coordinate contributions.