Closed jluethi closed 1 year ago
This behavior is consistent with how the copy task is defined. The timeline (in the observed behavior) should look something like this:
plate.zarr
), and of wells/images.mip
. This copies plate.zarr
into plate_mip.zarr
, and modifies in-place all entries of the component list with the replacement plate.zarr -> plate_mip.zarr
.plate.zarr
string is not any more present in the dataset metadata, and the input for the next task is plate_mip.zarr
. If I re-run a copy-ome-zarr task with suffix abc
, this will copy plate_mip.zarr
into plate_mip_abc.zarr
, and so on.In other words, we do not have a rich structure for component lists (like 2D vs 3D label), and at at the same time we would not currently able to "select" what the current components to work on are. Ref https://github.com/fractal-analytics-platform/fractal-server/issues/792, for instance, but this touches upon multiple issues related to the definition of tasks I/O, the constraints on workflow structure, the reserved arguments, ...
I think this is exactly what Joel's two helper tasks (the ones for 3D->2D->3D workflows) were doing, right? Having some "fake" task that is there only in order to update the dataset metadata in the desired way. Note that this may be possible also by modifying the dataset properties (*), but that would require a user's action (and then it would split workflow execution in two parts).
(*) I'm not sure of whether this is currently supported, but it'd be easy to add to fractal-server and to the clients if needed.
Hmm, yes, that makes sense. I guess it shows the limitation of our current component handling system and how overwriting components to get from 3D to 2D was always a bit of a hack.
Unfortunately, this breaks the core flow for rerunning workflows once they contain a copy-ome-zarr anywhere. We clearly have to address the general handling of components eventually.
What surprised me here is that I'm only reusing the same output dataset and rerunning the full workflow. I would have expected that to work, as the initial task does create a fresh, correct component list. I don't fully understand why it gets the component list of the output dataset instead of the one created by the Create OME Zarr task.
Let's briefly discuss afterwards
I'm trying this:
plate.zarr
and plate_mip.zarr
mip_mip
).At this point, I also observe plate_mip_mip.zarr
was created (by the copy-ome-zarr task, and in fact it contains no actual image data). The copy-ome-zarr logs read
2023-09-06 11:40:35,393; INFO; START copy_ome_zarr task
2023-09-06 11:40:35,393; INFO; list_plates=['/home/fractal-share/data/output2/20200812-CardiomyocyteDifferentiation14-Cycle1.zarr', '/home/fractal-share/data/output2/20200812-CardiomyocyteDifferentiation14-Cycle1_mip.zarr']
2023-09-06 11:40:35,393; INFO; zarrurl_old='/home/fractal-share/data/output2/20200812-CardiomyocyteDifferentiation14-Cycle1.zarr'
2023-09-06 11:40:35,393; INFO; zarrurl_new='/home/fractal-share/data/output2/20200812-CardiomyocyteDifferentiation14-Cycle1_mip.zarr'
2023-09-06 11:40:35,393; INFO; meta_update={'copy_ome_zarr': {'suffix': 'mip', 'sources': {'20200812-CardiomyocyteDifferentiation14-Cycle1_mip': '/home/fractal-share/data/output2/20200812-CardiomyocyteDifferentiation14-Cycle1.zarr'}}}
2023-09-06 11:40:35,403; INFO; Start open_zarr_group_with_overwrite (overwrite=True).
2023-09-06 11:40:35,403; INFO; Zarr group /home/fractal-share/data/output2/20200812-CardiomyocyteDifferentiation14-Cycle1_mip.zarr already exists, with keys=[]
2023-09-06 11:40:35,407; INFO; well_paths=['B/03']
2023-09-06 11:40:35,407; INFO; image_paths=['0']
2023-09-06 11:40:35,408; INFO; I will now read FOV_ROI_table from zarrurl_old='/home/fractal-share/data/output2/20200812-CardiomyocyteDifferentiation14-Cycle1.zarr', convert it to 2D, and write it back to the new zarr file.
2023-09-06 11:40:35,419; INFO; I will now read well_ROI_table from zarrurl_old='/home/fractal-share/data/output2/20200812-CardiomyocyteDifferentiation14-Cycle1.zarr', convert it to 2D, and write it back to the new zarr file.
2023-09-06 11:40:35,430; INFO; zarrurl_old='/home/fractal-share/data/output2/20200812-CardiomyocyteDifferentiation14-Cycle1_mip.zarr'
2023-09-06 11:40:35,430; INFO; zarrurl_new='/home/fractal-share/data/output2/20200812-CardiomyocyteDifferentiation14-Cycle1_mip_mip.zarr'
2023-09-06 11:40:35,430; INFO; meta_update={'copy_ome_zarr': {'suffix': 'mip', 'sources': {'20200812-CardiomyocyteDifferentiation14-Cycle1_mip': '/home/fractal-share/data/output2/20200812-CardiomyocyteDifferentiation14-Cycle1.zarr', '20200812-CardiomyocyteDifferentiation14-Cycle1_mip_mip': '/home/fractal-share/data/output2/20200812-CardiomyocyteDifferentiation14-Cycle1_mip.zarr'}}}
2023-09-06 11:40:35,430; INFO; Start open_zarr_group_with_overwrite (overwrite=True).
2023-09-06 11:40:35,430; INFO; Zarr group /home/fractal-share/data/output2/20200812-CardiomyocyteDifferentiation14-Cycle1_mip_mip.zarr does not exist yet.
2023-09-06 11:40:35,430; INFO; well_paths=['B/03']
2023-09-06 11:40:35,431; INFO; image_paths=['0']
2023-09-06 11:40:35,431; INFO; I will now read FOV_ROI_table from zarrurl_old='/home/fractal-share/data/output2/20200812-CardiomyocyteDifferentiation14-Cycle1_mip.zarr', convert it to 2D, and write it back to the new zarr file.
2023-09-06 11:40:35,442; INFO; I will now read well_ROI_table from zarrurl_old='/home/fractal-share/data/output2/20200812-CardiomyocyteDifferentiation14-Cycle1_mip.zarr', convert it to 2D, and write it back to the new zarr file.
2023-09-06 11:40:35,452; INFO; END copy_ome_zarr task
and the line which doesn't look right is
2023-09-06 11:40:35,430; INFO; zarrurl_old='/home/fractal-share/data/output2/20200812-CardiomyocyteDifferentiation14-Cycle1_mip.zarr'
Here is the explanation (in copy-ome-zarr):
# List all plates
in_path = Path(input_paths[0])
list_plates = [p.as_posix() for p in Path(in_path).glob("*.zarr")] # <--- THIS IS THE PROBLEM
logger.info(f"{list_plates=}")
meta_update: dict[str, Any] = {"copy_ome_zarr": {}}
meta_update["copy_ome_zarr"]["suffix"] = suffix
meta_update["copy_ome_zarr"]["sources"] = {}
# Loop over all plates
for zarrurl_old in list_plates:
zarrfile = zarrurl_old.split("/")[-1]
old_plate_name = zarrfile.split(".zarr")[0]
new_plate_name = f"{old_plate_name}_{suffix}"
new_plate_dir = Path(output_path).resolve()
zarrurl_new = f"{(new_plate_dir / new_plate_name).as_posix()}.zarr"
meta_update["copy_ome_zarr"]["sources"][new_plate_name] = zarrurl_old
... # copy zarrurl_old to zarrurl_new
We are currently copying all zarr files in the input folder.
Here is a first solution:
The metadata
input parameter already includes a list of plates (the value corresponding to the plate
key). At the very least, we should use that one rather than just listing the existing folders. We should still verify that each plate does exist in the input folder, but that's not enough as a criterion for including a plate in the copying procedure.
Any better idea?
Of course we can do better, e.g. we could make this one a plate-level task, but this would require some not-yet-available features:
Sticking with the current non-parallel version, we can make it more flexible with some glob patterns - but this would mostly be a temporary fix while waiting for more flexible architecture/tasks. Therefore I'd rather stick with a default behavior (and no new input parameter), and improve it as part of the planned future refactoring.
I have observed an unexpected behavior when rerunning workflows with the copy OME-Zarr tasks creates an additional level 20200812-CardiomyocyteDifferentiation14-Cycle1_mip_mip.zarr, then 20200812-CardiomyocyteDifferentiation14-Cycle1_mip_mip_mip.zarr when running a third time.
Expected behavior: I rerun the workflow, the content of the 20200812-CardiomyocyteDifferentiation14-Cycle1_mip.zarr gets replaced with the new content.
Observed behavior: Every rerun creates a new _mip addition to the OME-Zarr.
Not sure if this has to do with the dataset somehow? i.e. the way we update the components in the copy task
Example workflow (the registration tasks can also be skipped, just the 4 tasks are relevant): workflow-export-Multiplexing_registration-1693933995989.txt