Adds subflows to group tasks and speed up czi workflow - Githubissues

niaid / image_portal_workflows

Workflows related to project previously referred to as "Hedwig"

BSD 3-Clause "New" or "Revised" License

5 stars 1 forks source link

Adds subflows to group tasks and speed up czi workflow #381

Closed annshress closed 9 months ago

annshress commented 10 months ago

Closes #377

Changes

Separate bulky generate_zarr into three different funcs: bf2raw, rechunk, and copy-assets tasks
Add subflow for czi pipeline where each czi files heavy computations are handled in a new node
Add subflow to group common tasks for save and cleanup (for all workflows)
Adds copy_workdir_logs_to_assets. Refactor copy_workdir_to_assets to copy_workdir_logs_to_assets.

This PR doesn't introduce any:

[x] Binary files
[x] Temporary files, auto-generated files
[x] Secret keys
[x] Local debugging print statements
[x] Unwanted comments (e.g: # Gets user from environment for code os.environ['user'] )

This PR contains valid:

[x] tests