tcompa
commented
1 year ago As of #803, we introduced a server-side history. This does not necessarily need to fully correspond to the ome-zarr-side history, since:
- server-side we need to store several details that would not fit in the ome-zarr (e.g. the list of all the failed tasks, and also the IDs of all relevant WorkflowTasks)
- server-side history should not be organized according to OME-NGFF specs (AKA: it's a history based on which tasks were submitted and/or executed, e.g. without any per-image or per-well granularity)
With this first version of the history, we can answer part of the questions above:
1. What is written as history?
- Tasks that ran on the data -> ✅
- Parameters that were used -> ✅
- Granularity level: Per plate or per image? -> NA (per-workflowtask)
- The exported workflows(s)? -> ✅ (actually: more than that)
- (Logs?) -> ❌
- (The full actual data ;) ) -> ❌
2. Where is history written to?
- The database -> ✅
- The OME-Zarr file -> NA (not yet)
- A custom history file somewhere -> ❌
3. Who is in charge of writing the history?
- Each task -> NA (not yet)
- The
run_fractal_taskfunction NA (not yet) - The server -> ✅
4. When is history written?
- When a task is finished or fails
I'm starting a new overview issue for the discussions around storing history/status of what workflows were run on a given dataset. We had prior discussions on this topic in https://github.com/fractal-analytics-platform/fractal-server/issues/236, https://github.com/fractal-analytics-platform/fractal-server/issues/108, https://github.com/fractal-analytics-platform/fractal-server/issues/14, https://github.com/fractal-analytics-platform/fractal-tasks-core/issues/177#issuecomment-1432643699 and in some other places.
This issue shall be the overview of what decisions were made and link to all the relevant sub-issues
Motivation
For reproducibility, to allow easier debugging and potentially to allow smarter submission scenarios (rerun parts, rerun what failed etc.), we want to have history of what processing has been applied to which part of the dataset.
Main questions
1. What is written as history? => Issue https://github.com/fractal-analytics-platform/fractal-server/issues/507
My default would be information on which task was run with which parameters for each image
2. Where is history written to? => Issue https://github.com/fractal-analytics-platform/fractal-server/issues/508
I strongly favor writing it with the OME-Zarr file. We can discuss whether the database also keeps track of it, but ground-truth should be in the actual OME-Zarr file.
3. Who is in charge of writing the history? => Issue https://github.com/fractal-analytics-platform/fractal-server/issues/509
run_fractal_taskfunctionI'm unsure about this one
4. When is history written? => Issue https://github.com/fractal-analytics-platform/fractal-server/issues/511
Are there more general questions we need to address on this topic?