Tolerance for frame/sync pulse mismatch

[mabuice]

Describe the Issue A clear and concise description of what the QC issue is. We need to decide upon a tolerance threshold for the size of acceptable mismatches between the number of sync pulses and the number of eyetracking frames.

The current threshold is 15 more sync pulses than eyetracking frames, and 0 more eyetracking frames than sync pulses. This is an issue due to a known problem that has affected many experiments in which the sync signal was terminated before the eyetracking camera.

Example "negative dropped frame" numbers from TaskTrainedNetworksMultiscope are as high as 30, but sometimes only 1 or 2 and often around 10-15 more eyetracking frames than sync pulses.

Current behavior in the SDK is to truncate the sync pulse series by removing the extra pulses at the end and aligning the remaining arrays. Suggested behavior for negative dropped frames is to remove the trailing extra eye frames and align the remaining arrays.

Scope Most multiscope experiments from the past few months? This definitely affects all but one session of TaskTrainedNetworksMultiscope, which began collection in October of 2021.

[mabuice]

I should have mentioned that we also need a means of identifying the experiments affected by the known early sync termination.

[DowntonCrabby]

@mabuice do you believe there should be different tolerances for different data streams? For example: I would imaging that both the physio and the stimulus should have extremely low tolerances as those are the most important datastreams, but can the allowable tolerances be larger for the behavior monitoring video or the eye tracking videos?

[mabuice]

I guess my answer is 'yes', because I don't see how you can reasonably apply a single criterion to different data streams. Ideally, the tolerance should be "0", but known issues might cause us to change this. The early sync pulse termination is an example, where we know we should be able to trust the alignment, so all we need to do is truncate. In that case, we actually don't need any tolerance, we just need to know that early termination is what happened (unless a really large discrepancy is indicative of a different problem itself). To be honest, the 15 extra sync pulses makes me a little nervous, since I don't know why it's there or why it is applied to ALL data.

[DowntonCrabby]

I don't believe it's applied to all data. Here are our current tolerances:

Physio: microscope dependent: Nikon:1, Scientifica: 6 (really 15), Meso: none Eye tracking: 100 behavior monitoring: 200

The thresholds above mean that there are more sync pulses than video frames. I would have to dig in to whether these physio tolerances are actually up to date with what is in the code because I remember the 15 frame threshold as well, but the last documentation I have is reflected in what's above. It also obviously doesn't account for more video frames than sync pulses, which is an issue as well.

[mabuice]

Oh, maybe I'm confused, then. I'm talking about the following: https://github.com/AllenInstitute/AllenSDK/blob/2e17977b90dcb5a425bcf4258b91aa6168da5ede/allensdk/brain_observatory/behavior/eye_tracking_processing.py#L199

Everything in the behavior part of the sdk runs that function when it deals with the eyetracking streams, regardless of instrument (that I can tell). That's for the eyetracking, mind you, not physiology, so I'm not sure what the numbers you have above are referring to. What am I missing?

[mabuice]

Ah, I'm confusing qc operator applied metrics with what the sdk itself does. We should probably make sure these two systems are convergent in an appropriate way. :)

[DowntonCrabby]

12/13/21 ops meeting notes: We should separate this into two different issues:

• define the threshold at which we throw data out (eg. above 15 frame mismatch)
• how to do the alignment if mismatches are present but they are below the failure threshold

we still need to determine what the relationship should be between the thresholds in SDK and the thresholds in mouse-seeks. Some options are:

• mouse-seeks uses the "default" threshold applied to the SDK, but we leave an option for end-users to define a more strict threshold via the SDK

This discussion should go to the stakeholders group and might be a larger discussion on how we do timestamp realignment. Some questions we might have are :

• what are the possible failure modes
• what are the fixes(if any) for a specific failure mode
• what metrics identify which failure mode we are in?

should get stats on how frequently this occurs.

[mabuice]

Incidence numbers:

These are from all the sessions from the project codes listed below where there was at least one experiment with 'workflow_state==passed'. If the number of excess eye frames being 1 is due to the metadata frame, then the problem codes are VisualBehavior (14/326), VisualBehaviorMultiscope (1/216), LearningmFISHDevelopment (5/5), and TaskTrained (2/87), so we might be able to just fail these sessions and move on. Let me know if I should look more into anything not labeled "passed".

As a reminder, in the situations where the number of frame_times (sync pulses) is larger than the number of eye frames, the SDK will quietly truncate when the difference is smaller than 15, and throw an error otherwise.

question for @quinnlh: Are the situations where there is 1 extra eye tracking frame due to the "metadata" frame and how would we know this? (also, which frame should we delete in that case, the first or the last?)

['VisualBehavior', 'VisualBehaviorMultiscope', 'VisualBehaviorTask1B', 'VisualBehaviorMultiscope4areasx2d', 'MultiscopeSignalNoise', 'LearningmFISHDevelopment', 'TaskTrainedNetworksMultiscope', 'LearningmFISHTask1A', 'omFISHGad2Meso']
VisualBehavior
Total sessions:  326
Number of sessions where eye_data is longer than frame_times:  14
[12 25  7 24  9  5  5  8  6  4  6  6  2 15]
Number of sessions where eye_data is shorter than frame_times:  12
[-4 -2 -1 -5 -8 -5 -7 -3 -8 -4 -5 -2]

VisualBehaviorMultiscope
Total sessions:  216
Number of sessions where eye_data is longer than frame_times:  1
[18]
Number of sessions where eye_data is shorter than frame_times:  2
[-6 -9]

VisualBehaviorTask1B
Total sessions:  244
Number of sessions where eye_data is longer than frame_times:  0
[]
Number of sessions where eye_data is shorter than frame_times:  11
[    -2     -4 -14855    -14    -14     -5     -1     -3    -12     -4
     -7]

VisualBehaviorMultiscope4areasx2d
Total sessions:  131
Number of sessions where eye_data is longer than frame_times:  0
[]
Number of sessions where eye_data is shorter than frame_times:  0
[]

MultiscopeSignalNoise
Total sessions:  64
Number of sessions where eye_data is longer than frame_times:  0
[]
Number of sessions where eye_data is shorter than frame_times:  0
[]

LearningmFISHDevelopment
Total sessions:  5
Number of sessions where eye_data is longer than frame_times:  5
[6 6 6 6 9]
Number of sessions where eye_data is shorter than frame_times:  0
[]

TaskTrainedNetworksMultiscope
Total sessions:  87
Number of sessions where eye_data is longer than frame_times:  84
[15  1  1  1  1  1  1  1  1  1  1  1  1  1  1  1  1  1  1 10  1  1  1  1
  1  1  1  1  1  1  1  1  1  1  1  1  1  1  1  1  1  1  1  1  1  1  1  1
  1  1  1  1  1  1  1  1  1  1  1  1  1  1  1  1  1  1  1  1  1  1  1  1
  1  1  1  1  1  1  1  1  1  1  1  1]
Number of sessions where eye_data is shorter than frame_times:  3
[-12 -55 -40]

LearningmFISHTask1A
Total sessions:  26
Number of sessions where eye_data is longer than frame_times:  26
[1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1]
Number of sessions where eye_data is shorter than frame_times:  0
[]

omFISHGad2Meso
Total sessions:  14
Number of sessions where eye_data is longer than frame_times:  13
[1 1 1 1 1 1 1 1 1 1 1 1 1]
Number of sessions where eye_data is shorter than frame_times:  1
[-11]

[mabuice]

These are histograms of the interframe intervals based on the recorded frame times. "delta" refers to the number of excess eye frames than frame_times (sync pulses). "1" in the histogram values means that the interval was at or near either 1/30 or 1/60, accordingly. "2" means twice that and so on.

It appears (at this level) there is no obvious difference in the frame time distributions between delta=0 experiments and others, unless the 0's and 2's should be cause for alarm.

[DowntonCrabby]

@mabuice I don't know the state of this issue but I'm guessing it's resolved to some degree? If not I will keep the ticket open

[matchings]

@DowntonCrabby has this been resolved?