Split CPU and GPU operations into top-level pipelines

hubmapconsortium / codex-pipeline

CODEX data processing code

GNU General Public License v3.0

11 stars 4 forks source link

Split CPU and GPU operations into top-level pipelines #21

Closed mruffalo closed 3 years ago

mruffalo commented 3 years ago

This pipeline now contains a mix of CPU and GPU processing, and some of the CPU-oriented processing is memory-intensive also. All pipeline steps are executed from one top-level workflow, and all steps must be executed on a GPU node because of the Cytokit step.

Split this into three "top-level" workflows:

illumination correction and stitching stage 1
Cytokit
background subtraction and stitching stage 2

mruffalo commented 3 years ago

I made some (preliminary, untested) progress in https://github.com/hubmapconsortium/codex-pipeline/tree/mruffalo/split-cpu-gpu-pipelines -- this probably needs some work and I haven't tried to script what's required to call each sub-workflow independently in sequence.

mruffalo commented 3 years ago

Merged in b6baef08778ebe105a4dde5b873b76c2706f1f7b, released in v2.1.1.