Closed tcompa closed 11 months ago
Great summary of the new copy task!
or parallel at the plate level
True and running on the plate level is sensible. But we couldn't use plate level in the current server implementation (because it would look for the plate key in metadata/components). Thus, this may run without a parallelization level and e.g. take the name of the Zarr file (that is in the input_paths folder) as an argument).
NOTE 1: This functionality needs to happen per-image (even if it belongs to a higher-level task).
True. But I think everything can be done based on zarr metadata (both the pixel sizes as well as the dimensions in pixels are contained in the metadata). Thus, I would say a task that runs on the plate level and just loops over all images is fine.
NOTE 2: The well_ROI_table name would not be fully appropriate, because it assumes the well=image correspondence (which is true for Fractal, but not in general). Maybe we can find a more generic name (image_ROI_table, full_ROI_table, main_ROI_table, ...).
Agreed! Let's go with image_ROI_table
.
When acting on an existing OME-Zarr, is the import-ome-zarr task expected to modify it? That is, is it OK to add ROI tables to an existing OME-Zarr?
it OK to add ROI tables to an existing OME-Zarr?
I would say yes, at least for that first implementation. ROI table are core to the further processing in Fractal, so there needs to be a good way to add them.
Should this be optional? Maybe, but default to on => create OME-Zarr tables
Thinking through some scenarios: In what scenarios would this not work? If there is a read-only OME-Zarr somewhere. But then: Where would the Fractal workflow save any output (as it needs to go into the OME-Zarr) => there may be workflows that require some type of Copy OME-Zarr (or subset of it) before it's on storage that Fractal can write to.
In that case: Import OME-Zarr without writing ROI tables. Then copy OME-Zarr (subset?). Then add ROI tables (e.g. through import OME-Zarr again, but this time importing the newly created copy?) Or have the copy task gain that functionality?
Are there other reasons to modify an OME-Zarr during import?
Modifying metadata may actually be something we'd want to allow, e.g. to add the wavelength_id parameter to the Omero tables. Not sure whether we'll work well with OME-Zarrs without that one.
My conclusion:
This non-parallel task (or parallel at the
plate
level, provided we implement https://github.com/fractal-analytics-platform/fractal-server/issues/802) would take a single ome-zarrplate.zarr
as an input.Its goals (sorted by increasing complexity) are to:
metadata_update
output - to be then parsed by fractal-server and placed inDataset.meta
(soon to become something likeDataset.components
- TBD). NOTE: This functionality requires browsing through groups according to the OME-NGFF structure, and constructing the lists based on the zarr attributes. Ideally we should never useglob
or anything similar, but only proceed through zarr groups/subgroups/attributes.well_ROI_table
) for each ome-ngff image inplate.zarr
, by combining the pixel sizes and array size into a physical-unit ROI. NOTE 1: This functionality needs to happen per-image (even if it belongs to a higher-level task). NOTE 2: Thewell_ROI_table
name would not be fully appropriate, because it assumes the well=image correspondence (which is true for Fractal, but not in general). Maybe we can find a more generic name (image_ROI_table
,full_ROI_table
,main_ROI_table
, ...).Side-note: we currently have a fixture like:
for our tests. Together with #212, the current issue should aim at making it fully obsolete, since the relevant information would be gathered elsewhere.