Logger: Eye Tracking, Gaze

[bastianilso]

Since commit 70d2eeccc9ae4e9943403fb895dd7fb6e41081a8 we now have eye tracking - the remaining work will be to log it as part of Whack-a-mole's existing logging mechanism.

[bastianilso]

Update: Using Whack-a-mole's logger we can now log eye tracking information at Game Framerate (60FPS) in sync with game-related events, here is a sample:

GazeHitPoint: World Coordinate of a point on a surface which the direction of the gaze hits. GazeOrigin: World Coordinate of the point's origin (which is just the camera's coordinates). GazeConfidence: Confidence level in the eye tracking data that has to be reached before we update GazeHitPoint. GazeAngleErrorEstimate: How much angular error is present (ie. is the direction trustworthy, see implementation here). This is different from e.g. depth error (is the deepness trustworthy). GazeConfidenceTreshold: Minimum threshold used to discard noisy eye tracking data (it will not show up in the logs).

GazeHitPointX	GazeHitPointY	GazeHitPointZ	GazeOriginX	GazeOriginY	GazeOriginZ	GazeConfidence	GazeAngleErrorEstimate	GazeConfidenceTreshold
-0.2881	2.2585	6.6529	-0.4804	1.274	0.1405	0.8056	0.4567	0.65
-1.8037	2.3665	6.3691	-0.4804	1.274	0.1405	0.9627	0.4478	0.65
-1.4584	2.6274	6.4408	-0.4805	1.2741	0.1399	0.7235	0.4518	0.65
-1.8896	2.149	6.2239	-0.4806	1.2742	0.1397	0.4489	0.437	0.65
-0.9799	1.9561	6.4656	-0.4805	1.2743	0.1395	0.995	0.4423	0.65
-1.087	2.8643	6.3421	-0.4804	1.2743	0.1394	0.3214	0.4458	0.65
0.0824	2.2493	6.6712	-0.4803	1.2744	0.139	0.0423	0.4595	0.65
-1.2528	2.7381	6.3311	-0.4803	1.2745	0.1388	0.3096	0.4446	0.65
0.4977	2.3385	6.6285	-0.4802	1.2746	0.1386	0.283	0.4609	0.65
0.2312	2.4961	6.5194	-0.4802	1.2746	0.1383	0.853	0.4537	0.65

Remaining TODO:

• [ ] Whack-a-mole's Logger inherently sends data to both Online Database and CSV - A 1 minute gameplay takes around 2-3MB of logs. Do we want to disable sending the data to online database or is it fine (will be used for 15 mins so e.g. 45MB per session)?
• [ ] Will need to test whether we can log at e.g. 120Hz with this system - at 60FPS there is no visible performance impact, but when game finishes, it stalls for around 1 minute to process data.
• [ ] Will need to port our calculations of eye tracking in visual degrees.
• [ ] Get access to DepthError as described in PupilLabs documentation.
• [ ] Find out whether we can log GazeAngleErrorEstimate in a more sensible unit.

[bastianilso]

Here is some sample CSV data if one were to record the data using Pupil Capture. The data is generally much more rich, but you don't get the context of the game.

gazeposition.csv:

gaze_timestamp	world_index	confidence	norm_pos_x	norm_pos_y	base_data
784985.854014	0	0.004301002397621	2.48132857561395	-1.27608961474462	784985.854014-1
784985.857974	0	0.974754094354284	0.495539709514717	0.384694247353294	784985.857974-0
784985.861968	0	0.00484273104719	2.04913818428964	0.142221333781596	784985.861968-1

pupil_positions.csv:

pupil_timestamp	world_index	eye_id	confidence	norm_pos_x	norm_pos_y	diameter	method	ellipse_center_x	ellipse_center_y	ellipse_axis_a	ellipse_axis_b	ellipse_angle	diameter_3d	model_confidence	model_id	sphere_center_x	sphere_center_y	sphere_center_z	sphere_radius	circle_3d_center_x	circle_3d_center_y	circle_3d_center_z	circle_3d_normal_x	circle_3d_normal_y	circle_3d_normal_z	circle_3d_radius	theta	phi	projected_sphere_center_x	projected_sphere_center_y	projected_sphere_axis_a	projected_sphere_axis_b	projected_sphere_angle
784985.854014	0	1	0.004301002397621	2.48132857561395	-1.27608961474462	1489.36938351283	3d c++	794.025144196465	546.261507538709	1003.14367379997	1489.36938351283	-33.9420555362851	0.588058970113348	1	1	-0.894434567447696	4.26114504368639	11.5100751096792	12	0.314025857494805	0.105978734865329	0.317478612557254	0.100705035411875	-0.346263859068421	-0.932716374760161	0.294029485056674	1.21721067534592	-1.46324334320021	111.820520150105	349.530207397508	1292.78044306478	1292.78044306478	90
784985.857974	0	0	0.974754094354284	0.495539709514717	0.384694247353294	41.373091875511	3d c++	158.572707044709	147.67338063521	30.0050658756783	41.373091875511	77.2886236564509	4.82189581196307	1	2	-1.98986481812639	-4.40738363096985	81.325830064425	12	-0.157061172863936	3.26344252876161	72.2815359059463	0.152733637105205	0.639235513310955	-0.753691179873226	2.41094790598154	2.26430006449096	-1.37085639003334	144.829958867176	86.399631592612	182.967699047305	182.967699047305	90
784985.861968	0	1	0.00484273104719	2.04913818428964	0.142221333781596	1145.03011713395	3d c++	655.724218972685	205.866879892417	812.725779553822	1145.03011713395	-47.114920419092	0.588364087478392	1	1	-0.894434567447696	4.26114504368639	11.5100751096792	12	0.289065687049699	-0.032010789395457	0.366999226401719	0.098625021208116	-0.357762986090154	-0.928589656939789	0.294182043739196	1.20492510692386	-1.46498353269655	111.820520150105	349.530207397508	1292.78044306478	1292.78044306478	90
784985.865965	0	0	0.923728622532283	0.496110074160694	0.383671997057583	40.8986806856838	3d c++	158.755223731422	147.91872070618	29.5457382597477	40.8986806856838	77.1764127493941	4.76854347663404	1	2	-1.98986481812639	-4.40738363096985	81.325830064425	12	-0.135985024716599	3.2925150047486	72.3105823855618	0.15448998278415	0.641658219643204	-0.751270639905265	2.38427173831702	2.26745462387275	-1.36798529361004	144.829958867176	86.399631592612	182.967699047305	182.967699047305	90
784985.870441	1	1	0.003605410534564	2.29627705127639	-1.11934692217821	1444.91246000743	3d c++	734.808656408446	508.643261322771	1007.58051088914	1444.91246000743	-34.3780738496199	0.588436918477872	1	1	-0.894434567447696	4.26114504368639	11.5100751096792	12	0.289154362762825	0.09113760667241	0.320340741986126	0.098632410850877	-0.347500619751165	-0.932477863974422	0.294218459238936	1.2158920386583	-1.46541364376694	111.820520150105	349.530207397508	1292.78044306478	1292.78044306478	90
784985.874235	1	0	0.826595635567609	0.495357814829839	0.383599002237428	40.6843683452363	3d c++	158.514500745548	147.936239463017	29.4073441024191	40.6843683452363	77.3920498098804	4.74326119374303	1	2	-1.98986481812639	-4.40738363096985	81.325830064425	12	-0.164266654242757	3.29400011186058	72.3060806338869	0.152133180323636	0.641781978569202	-0.751645785878173	2.37163059687151	2.2676159912547	-1.37109404969265	144.829958867176	86.399631592612	182.967699047305	182.967699047305	90
784985.878279	1	1	0.006074164769612	2.00818866431976	0.106232785920658	1151.67790075583	3d c++	642.620372582323	214.504131379042	826.262852989081	1151.67790075583	-46.7833054920448	0.588326989931005	1	1	-0.894434567447696	4.26114504368639	11.5100751096792	12	0.27998860548621	-0.028198412157219	0.364570952007023	0.097868597744492	-0.357445287986967	-0.928792013139347	0.294163494965502	1.20526530094824	-1.46581181904944	111.820520150105	349.530207397508	1292.78044306478	1292.78044306478	90
784985.881976	1	0	0.907164986538651	0.495255794244261	0.384809427825705	40.9539737253706	3d c++	158.481854158163	147.645737321831	29.7235097296887	40.9539737253706	77.3637279665949	4.77268891871655	1	2	-1.98986481812639	-4.40738363096985	81.325830064425	12	-0.1678707357076	3.25921775567364	72.2757714600873	0.151832840201566	0.638883448886958	-0.754171550361475	2.38634445935827	2.26384233702057	-1.37212800585582	144.829958867176	86.399631592612	182.967699047305	182.967699047305	90
784985.886014	1	1	0.011135787096523	0.650071178037275	1.61066830913979	887.133150305704	3d c++	208.022776971928	-146.56039419355	647.014905248304	887.133150305704	-78.6054966310239	0.588311452342055	1	1	-0.894434567447696	4.26114504368639	11.5100751096792	12	0.045395330565399	-0.240810599399274	0.42634310504781	0.078319158167758	-0.375162970257138	-0.923644333719281	0.294155726171028	1.18622374680707	-1.48620504305545	111.820520150105	349.530207397508	1292.78044306478	1292.78044306478	90
784985.893994	1	0	0.994284456898967	0.494983014787817	0.384371417760179	41.3399906539116	3d c++	158.394564732102	147.750859737557	29.965574618376	41.3399906539116	77.4615093922401	4.81826587885273	1	2	-1.98986481812639	-4.40738363096985	81.325830064425	12	-0.177952197648459	3.27260200334182	72.2851003752616	0.150992718373161	0.639998802859305	-0.75339414076361	2.40913293942636	2.26529303440608	-1.3730001886728	144.829958867176	86.399631592612	182.967699047305	182.967699047305	90

world_timestamps.csv:

timestamps [seconds]	pts
784985.853642	0
784985.886975	2184
784985.920309	4368
784985.953642	6553
784985.986975	8737
784986.020309	10922

[bastianilso]

We ended up letting PupilCapture perform the data logging - it is faster, more precise/reliable and detailed.

We have a working screencasting and data logging pathway in whack-a-mole now (a recording of about 1 minute is around 250MB of data, FYI). Screenshot below is from Pupil Player, which is Pupil Labs' annotation/scrubbing software.

What remains is:

• [x] Automatically trigger recording start/stop when pressing "Play" and "Stop" buttons in-game. Not sure what we should do about "Pause"? (for now i'll just keep the recording going)
• [x] Research how we should parse PupilTime when raw data is exported from PupilPlayer
• [x] Provide a way to enable/disable the use of eye tracking in Whack-a-mole VR.
• [x] Log a PupilTime column in Whack-a-mole VR, as a passive event column, so we can synchronize the data easily.

[bastianilso]

Pull request #77 is now merged, closing this issue. Let's take separate/finer tasks related to eye tracking in their own respective issues.