Create structured records of certain characteristics of the input data and output data. For example, a summary (CSV? some sort of database?) of how many input points exist in each file, how many points are in each of the output files, average intensity or other summarizations of input properties etc. This could potentially also be useful when we start to develop better ways of handling the situation where the workflow fails halfway through the job. We need to track how much of the data we've already processed so that we don't need to start the entire workflow over again.
iannesbitt
commented
1 year ago