Move from linear execution strategy to modular block design

[gwaybio]

Currently, the recipe (as defined in jump-cellpainting/profiling-recipe#14) is linear, with each step progressing sequentially. For example, the normalization step happens before the feature selection step.

A use case came up in the JUMP project that we will want to apply different normalization steps to the same input file, and then process the same feature selection step to both normalization output files.

This process is akin to a block design, in which each pipeline step is performed if and only if a "block" is added to the yaml config file.

A couple of implications of this enhancement:

• We will need to perform a substantial refactor to introduce this change
• We should explore adding the execution steps to a workflow language.
• We will need to add functionality to specify the input file in each block.
• Explore adding dask to the mix to enable task-graph parallelization

[shntnu]

@gwaygenomics Do you recollect what you meant by "subprocess" above? Here are the recipe files, in case it helps point to things https://github.com/jump-cellpainting/profiling-recipe/tree/master/profiles

[gwaybio]

Ah, good point. I was wrong about this recipe using subprocess. We use it in the pooled recipe, so this is likely how my wires got mixed.