AllenInstitute / AllenSDK

code for reading and processing Allen Institute for Brain Science data
https://allensdk.readthedocs.io/en/latest/
Other
344 stars 149 forks source link

Move eyetracking into a github repo #2621

Open morriscb opened 1 year ago

morriscb commented 1 year ago

Currently, the code we run for eyetracking sits in a directory onprem rather than a version controlled repo. This epic encapsulates work for:

aamster commented 1 year ago

Eye tracking runs in 2 phases:

  1. Eye tracking using DeepLabCut: /allen/aibs/technology/waynew/eye/bin/eye_dlc_phase1.py

This uses a deep learning model to predict coordinates of parts of the eye. It predicts 12 points for the entire eye, pupil, and corneal reflection.

The config for the model is at /allen/aibs/technology/waynew/eye/universal_eye_tracking-peterl-2019-07-10/config.yaml. Various important data for the model can be found at /allen/aibs/technology/waynew/eye/universal_eye_tracking-peterl-2019-07-10 including training data, model, etc. EYE_TRACKING queue in LIMS

  1. Ellipse fitting /allen/aibs/technology/waynew/eye/bin/eye_dlc_phase2.py

This takes the coordinates output from (1) and fits and ellipse. We only ship the ellipse meta in the nwb files. EYE_ELLIPSE_FITTING queue in LIMS

  1. It can then run through an optional 3rd step, mapping the coordinates to monitor coordinates, using EYE_TRACKING_TO_SCREEN_MAPPING_QUEUE. This was not used for visual behavior. Wayne left a cryptic note that this module produced the wrong output and that it was not being used anymore. It was used for visual coding.

Here's an example eye tracking frame. For each frame, it predicts 12 points for eye, pupil and corneal reflection. The eye points are in green, pupil in red and corneal reflection in blue. It also includes the fitted ellipse center which is what we ship in the nwb file as a triangle for eye, pupil, and cr. (0, 0) is the top left. eye_tracking.png