Map handoffs and checkpoints for genome scale runs of image analysis workflow

[bethac07]

(Moved here from https://github.com/broadinstitute/pooled-cell-painting-analysis/issues/83)

Right now, the workflow has several stages and/or proposed stages

1. CellPainting illumination correction calculation
2. CellPainting illumination correction
3. CellPainting segmentation pipeline (see #81)
4. CellPainting stitching and splitting into tiles
5. Barcode calling illumination correction calculation
6. Barcode illumination correction application (see #82) and alignment.
7. Barcode color compensation and barcode calling "sanity check".
8. Barcode stitching and rescaling and splitting into tiles
9. Final profiling + barcoding pipeline

This creates 2 pre-setup steps, and at least 6 handoffs. Right now, each one is manual, with manual quality checks at each. For each one, we need to a) decide how we're going to do file handling and b) decide if and how we will determine success (quantitative cutoff? How/where do we check it? Manual visual inspection of something? Same thing) or if we think it can just proceed with something like an Amazon Lambda trigger.

• [x] Pre-1-2

• [x] Pre-5-6

• [x] 1 to 2

• [x] 2 to 3

• [x] 2 to 4

• [x] 3 to 9 - MANUAL

• [x] 4 to 9 - MANUAL

• [x] 5 to 6

• [x] 6 to 7

• [x] 7 to 8

• [x] 8 to 9 - MANUAL

OPTIONAL BUT REALLY NICE

• [ ] Post 7 (maybe to feed 8)
• [ ] 4-to-8 alignment check
• [x] 9 - setup and run
• [x] 9- cleanup

Misc notes

• For pre 1-2 and pre 5-6, trigger by upload of pipeline
• No check needed between 1 and 2 or between 5 and 6, ok to set auto-triggers
• 2 to 3 - in the illum correction application pipeline, which will be grouped by ~Plate~ EDITED 2020-05-18, Plate and Well, have it also calculate thresholds of cell segmentation stain and export that as a CSV. Then write a lambda function that (best case scenario) edits pipeline file with (50th? 75th? percentile) new threshold max to run a segmentation check pipeline, which runs on every 10th image and checks segmentation with and without a max (worst case scenario) emails you to run that pipeline. No auto-feed 3 to 9, should be manually checked. Ideally, triggered by either creation of the post-2 summary CSV OR a new pipeline being uploaded - CONFIRMED 2020-05-18
• 2 to 4 - automatic, triggered by lambda (specifically post-2 summary CSV).
• 4 to 9- should save 10x downsampled thing. No auto-trigger of 9, needs to be manually checked.
• 5 to 6 because of all the alignments is SLOW; it need to be batched as small as is practical, but while we are still using one file per well, batching to single files is less-than-ideal because loading of all files for a well takes 15-25 minutes for the first site from a well, and mere seconds for the one after that.
• 6 to 7 should be triggered either by 6 completing OR upload of a new pipeline, ideally - DONE June 11th
• 7 to 8- in an ideal world, auto trigger if % perfect greater than threshold value (70%?); if not doable, auto-trigger stitching then human does manual check for any plates that may need to be adjusted.
• 8 to 9- should save 10x downsampled thing. No auto-trigger of 9, needs to be manually checked.

[bethac07]

Right now, uploading the pipeline for step 2-3 causes an infinite recursion loop; should fix that.

[bethac07]

~Right now, uploading pipeline 7 triggers a new run, but we probably don't want to just go ahead and do that on initial uploading~ Actually this is fine, because it needs step 6 to run first to write something to the metadata file, so if it is uploaded before step 6 is run there's no issue.