Support including/running a V1 task within a V2 WorkflowTask

[tcompa]

EDIT: updated as of April 17

High-level goal: support running (a subset of) V1 tasks within V2 workflows.

Under which assumptions a V1 Task can be run within the V2 runner?

Machine-verifiable assumptions:

1. task.is_parallel = True
2. task.parallelization_level in [“image”, “well”, “plate”]

Assumptions that are enforced within runner:

1. Argument input_paths has a single entry
2. Argument output_path is the same as input_paths[0]
3. Argument metadata is empty
4. Output metadata are ignored

Assumptions that can only be verified by looking at the task code:

1. Task does not use metadata or output_path V1 arguments
2. Task produces no (relevant) output metadata.

How does a V1 task look within a WorkflowTaskV2?

1. WorkflowTaskV2.is_legacy_task = True (False by default)
2. WorfkflowTaskV2.task_legacy.is_v2_compatible = True (False by default) -> this require action by an admin, who sets the flag to True after a quick check
3. WorkflowTaskV2.type="parallel"

[tcompa]

Partly addressed within #1399

[tcompa]

Now tested with versions https://github.com/fractal-analytics-platform/fractal-server/pull/1424/commits/16424fcba313f6674b794f637f943a6579005ed1 and https://github.com/fractal-analytics-platform/fractal-web/pull/465/commits/1589c6f1194b64c8b795edf814575658cb540bd4.

I added a cellpose task from 0.14.3 to a V2 workflow, using other tasks from 1.0.0a3.

The workflow runs successfully:

The logs specify that the cellpose task was a legacy one:

2024-04-22 08:46:08,357 - WF1_job10 - INFO - Start execution of workflow "Workflow cardiac-test-partial-2"; more logs at /somewhere/Fractal/fractal-server/tests/data/example_server_startup/Artifacts/proj_0000001_wf_0000001_job_0000010_20240422_064608/workflow.log
2024-04-22 08:46:08,357 - WF1_job10 - DEBUG - fractal_server.__VERSION__: 2.0.0a11
2024-04-22 08:46:08,357 - WF1_job10 - DEBUG - FRACTAL_RUNNER_BACKEND: local
2024-04-22 08:46:08,357 - WF1_job10 - DEBUG - slurm_user: None
2024-04-22 08:46:08,358 - WF1_job10 - DEBUG - slurm_account: None
2024-04-22 08:46:08,358 - WF1_job10 - DEBUG - worker_init: None
2024-04-22 08:46:08,358 - WF1_job10 - DEBUG - job.id: 10
2024-04-22 08:46:08,358 - WF1_job10 - DEBUG - job.working_dir: /somewhere/Fractal/fractal-server/tests/data/example_server_startup/Artifacts/proj_0000001_wf_0000001_job_0000010_20240422_064608
2024-04-22 08:46:08,358 - WF1_job10 - DEBUG - job.working_dir_user: /somewhere/Fractal/fractal-server/tests/data/example_server_startup/Artifacts/proj_0000001_wf_0000001_job_0000010_20240422_064608
2024-04-22 08:46:08,358 - WF1_job10 - DEBUG - job.first_task_index: 0
2024-04-22 08:46:08,358 - WF1_job10 - DEBUG - job.last_task_index: 2
2024-04-22 08:46:08,358 - WF1_job10 - DEBUG - START workflow "Workflow cardiac-test-partial-2"
2024-04-22 08:46:08,359 - WF1_job10 - DEBUG - SUBMIT 0-th task (name="Convert Cellvoyager to OME-Zarr")
2024-04-22 08:46:12,177 - WF1_job10 - DEBUG - END    0-th task (name="Convert Cellvoyager to OME-Zarr")
2024-04-22 08:46:12,177 - WF1_job10 - DEBUG - SUBMIT 1-th task (name="Maximum Intensity Projection HCS Plate")
2024-04-22 08:46:14,122 - WF1_job10 - DEBUG - END    1-th task (name="Maximum Intensity Projection HCS Plate")
2024-04-22 08:46:14,122 - WF1_job10 - DEBUG - SUBMIT 2-th task (legacy, name="Cellpose Segmentation")
2024-04-22 08:46:25,821 - WF1_job10 - DEBUG - END    2-th task (name="Cellpose Segmentation")
2024-04-22 08:46:25,821 - WF1_job10 - INFO - End execution of workflow "Workflow cardiac-test-partial-2"; more logs at /somewhere/Fractal/fractal-server/tests/data/example_server_startup/Artifacts/proj_0000001_wf_0000001_job_0000010_20240422_064608/workflow.log
2024-04-22 08:46:25,821 - WF1_job10 - DEBUG - END workflow "Workflow cardiac-test-partial-2"

The args.json file for the cellpose task looks like

{
  "input_ROI_table": "well_ROI_table",
  "use_masks": true,
  "relabeling": true,
  "diameter_level0": 60,
  "model_type": "nuclei",
  "cellprob_threshold": 0,
  "flow_threshold": 0.4,
  "normalize": {
    "type": "default"
  },
  "min_size": 15,
  "augment": false,
  "net_avg": false,
  "use_gpu": true,
  "batch_size": 8,
  "invert": false,
  "tile": true,
  "tile_overlap": 0.1,
  "resample": true,
  "interp": true,
  "stitch_threshold": 0,
  "overwrite": true,
  "level": 2,
  "channel": {
    "wavelength_id": "A01_C01"
  },
  "output_label_name": "nuclei",
  "input_paths": [
    "/tmp/legacy1"
  ],
  "output_path": "/tmp/legacy1",
  "metadata": {},
  "component": "20200812-CardiomyocyteDifferentiation14-Cycle1_mip.zarr/B/03/0"
}

where we can identify the legacy arguments (metadata={}, component, I/O paths).

---

I have not observed anything unexpected. Closing.