Player/Capture: Allow preview of eye images with intrinsic matrix transformation

[gabrielDiaz-performlab]

I will soon be using your eye cameras to get an eye image through the optics of an HMD. I have learned from my grad student (Rudra on discord) that pupil player/capture is able to import eye camera intrinsic matrices / camera calibration files. It would be excellent if I could toggle the effect of these matrices during pupil detection, similarly to how I can preview the scene camera matrix transformation using your new camera calibration plugin (an excellent addition!).

[marc-tonsen]

Hi @performlabrit !

Thank you for the feedback! Your proposal makes sense given the fact that we have this exact feature for the scene camera as you correctly describe. There are however two reasons to why this is not available for the eye cameras as well, and to why the camera calibration plugin only supports calibrating the scene camera:

1) The camera distortion present in the eye images is very little. In order to do gaze estimation it is not necessary to add the additional computation required to perform undistortion.

2) The eye cameras record IR images and have a very small depth of field. This makes it very difficult to calibrate them with the approach used in the camera calibration plugin. You would have to use a very small calibration pattern and hold it very close to the camera. Things shown on a display or printed onto paper are typically not visible in IR light, so you would also have to come up with a way of making the small calibration pattern visible in the eye camera.

It is technically possible to make the eye cameras load intrinsics in the current implementation, but this is not recommended and not supported by the camera calibration plugin. Hence the option to toggle undistortion of the eye camera image on and off is missing.

Best, Marc

[gabrielDiaz-performlab]

Thanks for the input, Marc! I've got to push back a bit, though. Here are my responses to your own...

1) "There is little camera distortion."

My lab is currently trying to move the eye cameras behind the optics in the HTC Vive. The eye imagery will be distorted by both the mirror (likely this will be a simple linear compression along one or two axes), and by the barrel distortion of the optics, and by the distorted related to the use of a Fresnel lens (much more complicated, but perhaps we can ignore it). You could also use a matrix to account for foreshortening due to oblique placement of the camera off the optical path from the eyes to the screen, as in the current pupil labs configuration.

2) "It's not possible, or at least very difficult."

Depth of field is much better in the 200 Hz cams than the 120 Hz. Yes, a printed calibration grid is visible in IR, and a grid can be printed suitably small for placement in front of the camera. Trust me, or try it for yourself (Matlab's offers a camera calib. GUI).

[marc-tonsen]

Ok, I see that in your case distortion of the eye camera would be a significant problem and calibration would be valuable!

What type of printer are you using to print an IR visible pattern? So far I have only tried inkjet printers and with these I had little success. A laser printer might work better?

Although we will probably not explicitly add a calibration feature for the eye cameras, since we do currently not want to encourage people to calibrate their eye cameras if they use standard hardware, there is a workaround that might be handy for you:

The camera intrinsics calibration plugin currently only supports calibrating the scene camera, or more precisely the camera that is active in the main world window. You can however in the UVC Manager menu simply select an eye camera as your scene camera. Then you can calibrate it as if it were an actual scene camera including toggling undistortion on and off. The calibration will be saved and from then on loaded for the eye camera also if it us used in the eye camera window.

[papr]

@performlabrit Does this answer satisfy your needs? If yes, please close this issue.

[gabrielDiaz-performlab]

Yes, but I would like one more point of clarification. One of my students has looked into it and told me that there is no automatic way for the eye intrinsics to be applied it to the eye imagery. Is that true?

This is not a big deal because we can just apply it to the eye vids In an off-line manner, in Matlab.

On Mon, Oct 29, 2018 at 6:00 AM Pablo Prietz notifications@github.com wrote:

@performlabrit https://github.com/performlabrit Does this answer satisfy your needs? If yes, please close this issue.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/pupil-labs/pupil/issues/1320#issuecomment-433851839, or mute the thread https://github.com/notifications/unsubscribe-auth/AIi_20Vkz9V1goZ4wK3LimFK_H5tm6Iqks5uptG5gaJpZM4XUDPh .

--

Gabriel J. Diaz, Ph.D. Assistant Professor Rochester Institute of Technology Chester F. Carlson Center for Imaging Science

Founder of PerForM Labs Click for demos. https://www.cis.rit.edu/performlab/

Office 2108, Building #76 Rochester, NY 14623 Office: (585) 475-6215 gabriel.diaz@rit.edu

[papr]

there is no automatic way for the eye intrinsics to be applied it to the eye imagery

What do you mean by applying in this case? Do you want to undistort the eye images before pupil detection? In this case you will have to undistort images explicitly before calling this line: https://github.com/pupil-labs/pupil/blob/b1970a12f9f05be2b3a34eef3a30e4891cb5c836/pupil_src/launchables/eye.py#L571

[gabrielDiaz-performlab]

OK, thanks. I think you can close this thread.

On Mon, Oct 29, 2018 at 6:32 AM Pablo Prietz notifications@github.com wrote:

there is no automatic way for the eye intrinsics to be applied it to the eye imagery

What do you mean by applying in this case? Do you want to undistort the eye images before pupil detection? In this case you will have to undistort images explicitly before calling this line: https://github.com/pupil-labs/pupil/blob/b1970a12f9f05be2b3a34eef3a30e4891cb5c836/pupil_src/launchables/eye.py#L571

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/pupil-labs/pupil/issues/1320#issuecomment-433861323, or mute the thread https://github.com/notifications/unsubscribe-auth/AIi_2222_KKVzMfeiMeXj2WzB-UwIzUqks5uptk8gaJpZM4XUDPh .

--

Gabriel J. Diaz, Ph.D. Assistant Professor Rochester Institute of Technology Chester F. Carlson Center for Imaging Science

Founder of PerForM Labs Click for demos. https://www.cis.rit.edu/performlab/

Office 2108, Building #76 Rochester, NY 14623 Office: (585) 475-6215 gabriel.diaz@rit.edu