search

AllenInstitute / AllenSDK

code for reading and processing Allen Institute for Brain Science data
https://allensdk.readthedocs.io/en/latest/
Other
333 stars 149 forks source link

refactor sync code to have single source for line labels #1465

Open wbwakeman opened 4 years ago

wbwakeman commented 4 years ago

There are at least three places in the AllenSDK code base that define the "line labels" used in different versions of the sync.h5 files that are provided with BrainTV experiments and used to align the various data streams from these experiments. None of them are versioned, although we have been promised that this will happen with the next major iteration of the MPE changes to the systems that create these files.

This issue is to refactor all the places that define these lines into a single place that captures all of the variations of each line label.

Some background on the problem is provided here:

some EYE_TRACKING_TO_SCREEN_MAPPING jobs are failing because they do not recognize the line label 'eye_tracking'.

This function https://github.com/AllenInstitute/AllenSDK/blob/master/allensdk/brain_observatory/gaze_mapping/__main__.py#L29

uses https://github.com/AllenInstitute/AllenSDK/blob/master/allensdk/brain_observatory/sync_dataset.py#L92 to define labels for EYE_TRACKING_KEYS:

EYE_TRACKING_KEYS = ("cam2_exposure", # clocks eye tracking frame pulses (port 0, line 9) "eyetracking") # previous line label for eye tracking (prior to ~ Oct. 2018)

This is different than LIMS ophys processing, where OPHYS_TIME_SYNC jobs https://github.com/AllenInstitute/AllenSDK/blob/master/allensdk/internal/pipeline_modules/run_ophys_time_sync.py#L163 call OphysTimeAligner https://github.com/AllenInstitute/AllenSDK/blob/master/allensdk/internal/brain_observatory/time_sync.py#L181 which calls get_keys to determine the right set of keys to use based on the versions defined in that same file https://github.com/AllenInstitute/AllenSDK/blob/master/allensdk/internal/brain_observatory/time_sync.py#L21L51

I am still assuming that we can believe these two pages as credible documentation for what the various rigs are producing: http://confluence.corp.alleninstitute.org/display/SSCILABS/Ophys+pipeline+line+assignments+and+labels http://confluence.corp.alleninstitute.org/display/ENG/Neuropixels+Pipeline+Line+Labels

Practically, I think we are seeing three different labels in use for eye tracking in sync files: "eyetracking", "eye_tracking", and "cam2_exposure".

MPE has offered to rename labels for any files that I identify as wrong. At this point, I am thinking two things though. We should support each of those three labels for time_sync and for gaze_mapping Is there any reason gaze mapping does not use the same mechanism as ophys_time_sync to determine the keys to use? Same question for this other function that appears to be another place where eye tracking sync line labels are independently defined: https://github.com/AllenInstitute/AllenSDK/blob/master/allensdk/brain_observatory/behavior/sync/__init__.py#L169

wbwakeman commented 4 years ago

MPE is working up a set of VERSION_3_KEYS, which will also include an explicit, non-ambiguous way to determine that the keys are the version_3 set of keys and not something else. Will update with the keys when I have them.