Simple motion-correction problem app

[djkapner]

Labelers reported that the ROI labeling app is their best way to spot motion correction problems, and they like the Sagemaker app for assessing.

• [x] Create bare minimum Sagemaker Labeling Job to label "motion problem" or "pass"
  • app uses existing 350 experiments/full videos on s3 from ROI labeling jobs.
  • app uses a pre-made AWS video classification task. No mods.
• [ ] Collect feedback on if this bare minimum is sufficient.
  • important point: how long a video time span is needed. If the science team really wants to use this method for all 1600 release experiments, identifying a minimum time span for labeling these will greatly reduce the amount of processing and disk space needed to make these video artifacts. Suggestion: 100 seconds.
• [ ] If necessary, make custom tweaks to the video app based on feedback.
• [ ] Create artifacts for release experiments.
• [ ] Upload artifacts to s3 and launch job (require no consensus, just 1 vote per experiment).
• [ ] Provide labels to science team.... @matchings what would happen at this point?

cc: @wbwakeman

[djkapner]

@DowntonCrabby @nataliaorlova @matchings feel free to add feedback here. Email also ok.

Feedback:

• Kate: too slow. 8x-16x default speed.
• Natalia: add a 3rd classification option: low SNR. (note: I think you'll only be able to choose one)

[kschelonka]

For the slowness, it would be pretty easy to add options to speed up the video. Or you can just adjust playbackRate attribute on the video if you want it to always be 16x or whatever.

[matchings]

I agree with Kate that we need to be able to view the movie at 8x minimum. 16x or 32x may be ok as well. We are currently investigating whether 100x is acceptable but initial intuition suggests that is too fast.

we definitely need the full movie over the entire session, because motion correction failure can happen anytime during the session. we need to be able to scroll through the movie and jump to different time points.

we do not need the full image, although it is preferable to have it. one quadrant of the video would be acceptable if it helps with storage issues.

we can probably live without segmentation, although it is helpful to have a static fiducial on the image to see things moving around relative to something fixed. i dont think this is necessary, if it is too complicated to implement.

we absolutely need the experiment_id to be visible in the viewer, and for the experiment_id of cases flagged as problematic ("residual motion" vs "no residual motion") to be automatically saved and tracked.

ideally, the xy motion correction values that are output by the motion correction algorithm would be displayed, similar to how the ROI traces are shown in the ROI labeling app. this would allow us to 1) fast forward to portions of the movie that correspond to times where a lot or a little correction was applied to see whether there is residual motion, and 2) to figure out whether visually apparent residual motion in the movies corresponds to a discernible signature in the xy correction values that we can use to define a quantitative QC metric.

low SNR is another feature of the data that multiple annotators noticed during the labeling process and wished they could have documented for the displayed experiment. in addition to documenting residual motion, it would be very useful if we could also provide an indication of whether the experiment had poor SNR or not, to aid in our development of an appropriate SNR metric. (i.e. in addition to "residual motion" and "no residual motion" button options, there could be a "poor SNR" button that can be checked and tracked for future investigation)

[matchings]

@DowntonCrabby @nataliaorlova any comments or additions to what i described above?

[djkapner]

Heard from Marina that Doug may have something that works well enough for this week's QC-a-thon. Will put this back in sprint backlog and hear how that goes.