Update tasks to support Fractal V2

Update input API & output handling for:

Simple updates:

[x] Cellpose segmentation: ParallelTask, does not generate outputs
[x] Apply registration to image: ParallelTask. Input type is registartion=False, sets registration=True flag.
- [x] Implement overwrite_input=False functionality, return the relevant image list update
[x] Illumination Correction: Parallel Task.
[x] napari workflow wrapper: ParallelTask, does not generate outputs

Refactoring required

[x] Convert CellVoyager to OME-Zarr: New Compound Task that combines Create OME-Zarr + Yokogawa to OME-Zarr
[x] Convert CellVoyager to OME-Zarr multiplex: New Compound Task with a different init from regular one. Maybe combine the 2? Likely not.
[x] Maximum intensity projection: New Compound task together with Copy OME-Zarr, needs new handling for Task output
[x] Import OME-Zarr: Non-Parallel Task. Future work to handle existing image list? Needs to provide correct new task output
[x] Calculate Registration Image based: Newly a compound task
[x] Find Registration Consensus: NonParallel or Compound task, based on Apply registration to ROI table. Needs to compute consensus on well level, either in compound mode or within a non-parallel task. Make this compound to run on well level so that we have an example of that logic

Update Manifest

[x] Apply registration to image: ParallelTask. Input type is registartion=False, sets registration=True flag.
[x] Illumination Correction: Parallel Task. Input type is illumination_corrected=False, sets illumination_corrected=True
[x] Convert CellVoyager to OME-Zarr: New Compound Task

Testing

[x] Update & reenable manifest validation in Github CI
[x] Update & reenable test_valid_args_schemas
[x] Update test_task_interface in test_valid_task_interface once new Manifest is built
[x] Activate test_MIP_subset_of_images in test_workflows again: When MIP task is also ready (because it relies on metadata)
[x] Activate test_workflow_with_per_well_labeling_2D in test_workflows_cellpose_segmenation again (relies on MIP)
[x] Activate test_multiplexing_MIP in test_workflows_multiplexing again (relies on MIP)
[x] Turn back on test_multiplexing_registration in test_registration once multiplexing tasks are updated
[x] Activate test_import_ome_zarr_plate in test_import_ome_zarr again once Import task is updated

Future goals

[ ] Add tests for converting 2D data and check that the compute task assigns the correct is_3D=False
[x] Allow Illumination correction with overwrite=False
- [ ] Illumination correction testing: Also test overwrite=False
[x] Update testing helper functions:
- [x] prepare_3D_zarr and prepare_2D_zarr in _zenodo_ome_zarrs still return zarrs with the old metadata logic (defined in the zenodo_zarr_metadata fixture)
[ ] Illumination Correction with Channel parallelisation: Test this if time available
[ ] Test the multi-plate case of create_cellvoyager_ome_zarr_init: It's been there before, not sure how well tested it is
[ ] Update examples to use new API
[ ] Less testing whole workflows and more testing standalone tasks: Create fixtures that create the relevant inputs for many other tasks, create the image list metadata they need [avoid having image list in task CI as much as possible, besides checking they provide the correct new outputs)

I'm doing some refactoring work on the Create OME-Zarr & Yokogawa to OME-Zarr handling. At first, I thought I could just pass the patterns list between the two functions. But the regex for finding individual channels could in some cases break in that approach. Thus, I've now set up an init_args model like this one:

class InitArgsCellVoyager(BaseModel):
    """
    Arguments to be passed from cellvoyager converter init to compute

    Attributes:
        image_dir: Directory where the raw images are found
        plate_prefix: part of the image filename needed for finding the 
            right subset of image files
        well_ID: part of the image filename needed for finding the 
            right subset of image files
        image_extension: part of the image filename needed for finding the 
            right subset of image files
        image_glob_patterns: Additional glob patterns to filter the available 
            images with
        acquisition: Acquisition metadata needed for multiplexing
    """

    image_dir: str
    plate_prefix: str
    well_ID: str
    image_extension: str
    image_glob_patterns: Optional[list[str]]
    acquisition: Optional[int]

And pattern parsing like this:

        patterns = [
            f"{init_args.plate_prefix}_{init_args.well_ID}_*{A}*{C}*."
            f"{init_args.image_extension}"
        ]
        if init_args.image_glob_patterns:
            patterns.extend(init_args.image_glob_patterns)

fractal-analytics-platform / fractal-tasks-core

Update tasks to support Fractal V2 #669