Pupil Player 2.0.16 Windows 10 Surface Detection Errors

[VisionResearchBlog]

I am not sure if this is a bug or not but I am having trouble understanding the output from the surface tracker present in the exported file 'gaze_positions_on_surface.csv' When I plot the hits on the target they are under- and over-estimated compared to annotation by hand of the exported world.mp4 video. I've read the documentation on surfaces and believe I'm following best practices: https://docs.pupil-labs.com/core/software/pupil-capture/#surface-tracking

I am trying to understand how the surface tracker determines a hit when comparing the eye gaze position versus the surface geometry.

Is the current eye position pixel location used to determine a hit? Or or is a region around the eye position used (e.g. the Gaze circle in the 'Vis Circle' plugin)?

I assume the 'Min Marker perimeter' under marker detection is not use for classifying hits - Is this right?

Further, for each surface I assume Width & Instruction are only for stretching the 'Surface in Window' and not used to determine surface hits?

The documentation mentions a blog post mentioned but the link is dead https://pupil-labs.com/news/0392-release -- So I am not sure if there is a more in depth discussion to detail how hits are determined and any parameters to adjust. I looked through some of the github repository - https://github.com/pupil-labs/surface-tracker - but did not see any documentation.

I've attached a graphic that shows (bottom) raw output from the surface tracker (middle) segmented raw output (looking for streaks of surface hits at least 100ms in duration) (top) hand annotation using the exported world.mp4

I am uploading a sample session now to share with info@pupil-labs.com

Here is an example of on_surf showing hits on the surface that didn't happen (time 8:00 to 8:30) - Note the marks are not perfectly aligned because they are are in a different time base but are very close:

Here is an example of on_surf missing hits on the surface that did happen (time 2:00 to 5:00):

I've annotated 6 other files and they all have similar discrepancies (although this one is the most pronounced). When I watch the surface tracking in pupil player I see that it is off sometimes but usually pretty good when all markers are central (the wide angle does warp on the edges). I realise that sometimes false positives can occur if someone is holding something in front of the surface and looking at that instead but this doesn't seem to be happening that often. I am confused as to how the data from 'gaze_positions_on_surface.csv' is so mismatched with what I see in the exported world.mp4

[pfaion]

Hi @VisionResearchBlog

we will have a look at your recording next week, but maybe this helps already:

The visualization "lines" of your surface you see in Pupil Player are not accurately representing the outline of a rectangular surface. You can easily see this if you record a piece of paper (or any rectangular surface) and compare the real-world outline of the object with the visualization outline in Player: the "real" outline of the surface will be rounded, due to camera lens distortion and will therefore not match up perfectly with the straight outline of the visualization.

The visualization of the surface outline acts only on the distorted image that you see in Player. However, for matching the gaze to the surfaces, we use the undistorted image available in the background. Especially far away from the corners (e.g. in the middle between two corners) this effect will be large, so you might see a gaze point which is outside of the visualization outline, but is marked as inside of the surface in the export. In the optimal case it should then also still be inside the "actual" outline of the real-world surface that you recorded.

Please see the attached screenshot, the red point seems to be outside of the surface (visualization), but should be recognized as inside the surface in Pupil in the undistorted image.

Could this effect explain the differences you are seeing? You should not see any erroneous data close to the corners of the surface.

[VisionResearchBlog]

Hi @pfaion thanks for your reply! The case you describe for 'false positives' being adjacent to the edge of the defined surface makes sense, however I do not think this is what is happening. In the project I sent if you view the video from 8:00 to 8:30 (world index 14262 to 15200) you will see the eye never comes close to the surface. However, in the data file (rows 68994 to 76492) you will see that there are 713 true entries - I cannot understand how these detections are occurring

I looked closer at the data file for the second issue I mentioned with false negatives. I do not know why but the exported file 'gaze_positions_on_surface_Instruction.csv' has a gap in time between rows 34782 & 34783. If I am not mistaken the gaze_positions_on_surface files only detail frames where the surface is detected and omit others, so its like the surface was not detected at all. I re-ran the surface export without changing settings and the new file no longer had the gap. So I think the issue wasn't false negatives but just a bad export. I will rerun the export on other files if I notice a similar discontinuity or other examples of false negatives that are not fixed by re-running export

[VisionResearchBlog]

Also as a feature request it would be helpful if (1) the onscreen video in pupil player displays the name of any surface==true to see the detection results in realtime; (2) have an option to output the overlay of the surface ploygon to the exported world.mp4

[papr]

The visualization of the surface outline acts only on the distorted image that you see in Player. However, for matching the gaze to the surfaces, we use the undistorted image available in the background.

The iMotions exporter plugin exports the undistorted video using the available camera intrinsics stored in the recording. Looking at that export, it becomes clear quite quickly that the stored intrinsics do not fit your lens well as the undistorted video still includes a lot of distortion on the edges.

This also affects surface tracking and mapping gaze to the surface.

If you still have access to the headset, you can generate better intrinsics using the Camera Intrinsics Estimation plugin in Capture. It will store the intrinsics to <camera name>.intrinsics in pupil_capture_settings. Copy this file, rename it to world.intrinsics and replace the corresponding file in your recordings (back up the old intrinsics beforehand). Afterward, reopen the calibration and rerun the surface tracking. This should give you better results.

It might be necessary to delete the surface marker cache first before reopening the recording.

[VisionResearchBlog]

Hi @papr thank you for this response and explaining about the distorted vs. undistorted image. I will run the camera intrinsic estimation and re-evaluate and let you know how surface tracking works.

[papr]

In Capture, you can preview the undistorted image in real time. All edges that are straight in real life need to be straight in the undistorted image as well.

[VisionResearchBlog]

Hi @papr I can confirm that using a newly calibrated world.intrinsics file does solve most of the problems. I have found that by filtering to smooth the surface intersection data, segmenting into sections >100ms and concatenating segments that are within 150ms seems to look pretty good compared to the human rater. However, I am still seeing some false positives that I cannot explain. I will send a link to an example project in an email. In that project you will see that the imotions export looks like all distortion is corrected. However if you go to frame 1086 in PupilPlayer and then examine the output in 'gaze_positions_on_surface_Instruction.csv' you'll see several surface hits despite the point of gaze cursor not being on the surface. This participant's track is not very stable so I am wondering if maybe this is related? The eye images show that the track is not great but the cursor is not on the surface so not sure if they are related

[papr]

This participant's track is not very stable so I am wondering if maybe this is related?

When looking at the exported data, one can see that all false-positive "hits" have very low confidence. It is suggested to filter gaze data prior to further processing.

By default, Pupil Player only visualizes and processes gaze with a confidence of 0.6 or higher.