Eye tracking using a normal webcam

[zelmon64]

Hi @RalfCT,

I revisited my gaze mouse that uses OpenFace. While it certainly lacks the precision of IR eye trackers to put the mouse where you look, I figure it can be used to move the mouse in the direction that you look off-screen. This is like using the mouse keys and seems to work reasonably well. The added benefit of this method is that the mouse isn't wiggling all over what you read.

This isn't quite ready to be used yet as it can't click. I'm trying to make it so that looking at a point closer than the screen sends a click but it's being awkward. I'm avoiding using facial cues since that won't work for your friend and obviously dwell clicking won't work with this method.

If you'd be interested in trying this I can send you a copy once it's ready.

[RalfCT]

@zelmon64 My friend has almost no head movement but let's have a look at what you have to offer, it may be suitable elsewhere.

[zelmon64]

@RalfCT my gaze mouse uses no head movement and in fact currently it really doesn't like it if the head does move.

Here's the current version - it still doesn't click but I think it's a good demonstration of what it will be capable of.

https://drive.google.com/open?id=0B7SYYOqNW4FSd3lTRlhzWDBLVUk

After launching FaceLandmarkVid.exe two windows will appear: one with text and the other a reconstruction of a face if it can see one. The reconstruction can be on top of the video feed but I currently have that off to make the reconstruction clearer. If no face appears when you know it can see you press q to quit and try again.

When you have positioned the camera to view your face you then press m to start the calibration for mouse control. During this calibration look at the top, bottom and sides of the screen and don't blink. This calibration sets the gaze "dead-zone" and other parameters. Once complete you can either press m again to finish or wait for it to timeout and it will sound three beeps to let you know that it has. Press m any time to re-calibrate.

After calibration it enters mouse control mode. In this mode looking off screen should cause the mouse to move in that direction. Movement speed is proportional to how far off the screen you look and is limited to a short amount of time so that it can't run away for whatever reason. It will beep to indicate that it has stopped moving.

A beep is currently also triggered when the click action is performed to test how reliable it is. This is the part where you look at a point closer than the screen is - basically go crossed-eyed but such an extreme shouldn't be necessary unless the screen is fairly close to you. To exit press q.

This is only a proof of concept version but if you have the time I would appreciate any feedback you may have (including if you think it's a hopeless idea and I should give up on it). If needed I can try to make a video to show how to use it.

If I've forgotten to mention something - sorry.

[zelmon64]

Hi @RalfCT, I still haven't added mouse clicking just yet but I've added a magnification step to the click action. The idea is that to click somewhere the screen is first magnified around that area to increase precision. The second action then unzooms and would send the click.

To run the program replace the previous version with it. https://drive.google.com/open?id=0B7SYYOqNW4FSNTdlcEdkdGd3YjA

I also added keys 1, 2 and 3 to apply x1, x2 and x3 magnification and the space bar now does the same as m.

[RalfCT]

@William, I get a opencv_world310.dll missing error when trying to launch the program.

[zelmon64]

@RalfCT I didn't repackage opencv_world310.dll or any other of the dependencies with OpenFace_Gaze_Mouse_v00.01.01 as they are the same as what's included in OpenFace_Gaze_Mouse_v00.00.01. Just either overwrite the old FaceLandmarkVid.exe with the new one or copy across all the other files.

[RalfCT]

@William, I can't seem to find the OpenFace download.

[zelmon64]

@RalfCT The link was in this post and here they both are:

OpenFace_Gaze_Mouse_v00.00.01: https://drive.google.com/open?id=0B7SYYOqNW4FSd3lTRlhzWDBLVUk OpenFace_Gaze_Mouse_v00.01.01: https://drive.google.com/open?id=0B7SYYOqNW4FSNTdlcEdkdGd3YjA

Just so you know, putting "@ William" doesn't reference me but someone else. For me put @zelmon64 as you had been doing.

[RalfCT]

@zelmon64, sorry, no idea why I've been using "@ William". I'll download and play around with the software.

[RalfCT]

@zelmon64, probably because your emails come from William.

[RalfCT]

@zelmon64, I've downloaded the first version and extracted. When I run I immediately get a FaceLandmarkVid.exe has stopped working dialog, even when running as Administrator. Log file attached. Log.txt

[zelmon64]

@RalfCT that log doesn't really mean much to me. Could you please look at the properties of the dll files and make sure they're not blocked.

Are you able to run FaceLandmarkVid.exe from the latest release OpenFace_v0.2.3? It might be that you need some VS runtimes.

Edit: looking here you need the Visual C++ redistributable package.

[RalfCT]

@zelmon64, just tried to install the OpenFace version above, same 'FaceLandmarkVid.exe has stopped working' dialog. Trying to install the Virtual C++ package results in a 'Setup Failed' error. Since my OS is 64-bit I downloaded both 64-bit versions (OpenFace and Virtual C++).

[zelmon64]

@RalfCT that seems rather strange. I'm on win10 x64 and have both the x64 and x86 packages installed. Maybe try the x86 package or perhaps some of these suggestions.

[RalfCT]

@zelmon64, tried clean boot, resetting Installer Service and deleting Temp contents. Same problem. I'll have to leave this for another time.

[RalfCT]

@zelmon64, it seems I need to install Visual Studio 2015 and 'compile' the binaries before they will run. To be honest, I can compile anything mechanical, but am an absolute novice when it comes to programming. Although this still doesn't explain why Visual C++ won't install.

[zelmon64]

@RalfCT you shouldn't need to compile anything to run the .exe programs in the official release or what I posted. Compiling is what makes the .exe files. The whole point with sharing the .exe files is that other people are suposed to be able to use them straight away without needing any knowledge of how to compile from source.

I'm not sure why the Visual C++ redistributable package is giving you so much trouble. In the past when I've needed it and also when I've shared other programs and othe people have needed to install it, it has always simply installed and then the programs could be run.

[RalfCT]

@zelmon64, okay, I seem to have found part of the problem. I must've previously installed a 32-bit version of Virtual C++ bundled with one of the software packages. I can now run OpenFace 32-bit FaceLandmarkVid.exe but still not your version of the file (still get the 'Program stopped working' error).

[zelmon64]

@RalfCT that still seems strange because as I said, I have both the x64 and x86 Visual C++ redistributable packages installed. I'm assuming "Virtual C++" is just a typo and you haven't been trying to use some other obscure package.

Here's my program built for 32-bit (I included the dependencies): https://drive.google.com/open?id=0B7SYYOqNW4FScGV5VTdsVEtvMk0

[RalfCT]

@zelmon64, I can't believe it, running your 32-bit build I get a "The procedure entry point MagSetFullScreenTransform could not be located in dynamic link library MAGNIFICATION.dll' error. "Visual" above should be Virtual.

[zelmon64]

@RalfCT what version of Windows are you on? Magnification.dll should be supported on Windows Vista and later. I can take the magnification back out but it won't be as good.

Edit: it's supposed to be located in: "C:\Windows\System32\Magnification.dll" and "C:\Windows\SysWOW64\Magnification.dll"

Edit2: it seems that MagSetFullScreenTransform is only supported on Windows 8 and later. I'll try to make a build using the not-full-screen magnifier.

[RalfCT]

@zelmon64 I'm running the above on a laptop: Windows 7 Pro. 64-bit Service Pack 1. Magnification.dll is in the above System folders.

[zelmon64]

@RalfCT Windows 7 that's the problem then. Here's a build with no magnification but the mouse becomes bound to the region that would have been magnified.

https://drive.google.com/open?id=0B7SYYOqNW4FSRUJKSXJvd0RLS0U

[RalfCT]

@zelmon64, no. now I get a "Program can't start because opencv_world310.dll is missing..." error. Unless I should copy the above FaceLandmarkVid over the one in the 32-bit build folder? Okay, we have action, I renamed FaceLandmarkVid in the 32-bit folder and inserted the above version. Video is in B&W, is that normal?

[zelmon64]

@RalfCT sorry I forgot to put that you needed to copy over the dependencies - same again with this next one. That version was supposed to be in B&W and the next one is in colour. I haven't decided what's best: B&W, colour or just the landmarks.

I managed to use the windowed magnifier in this version. It's not quite as nice as the full-screen version but it seems good enough (and will hopefully work for you).

https://drive.google.com/open?id=0B7SYYOqNW4FSQ2xzRE82cHpJMk0

[RalfCT]

@zelmon64, I played around with the software last night but couldn't get it to work consistently. The mouse would move, but not always in sync with my eye movements. While OpenFace may be a suitable face identification tool due to the larger area measured, it seems to struggle to interpret where the eyes are looking. For precise pointing an eye-tracker may still be the best option. Nonetheless, keep me updated of new developments.