Talking with @davidalexandershapiro, he gave some interesting requirements for his flows:
Whenever a file is created, the file (in parrallel):
copied to NERSC
run through a low-res reconstruction on a beamline machine
When the file is finished copying to NERSC
A high-res reconstruction is performed
The low-res reconstruction lets the user quickly get some feedback on the success of the scan. The high-res will be used for actual research.
This introduces a few new concepts for his flows.
The main flow will invoke another flow, but not as a sub-flow. It will instead launch a new flow from a deployment so that it can get picked up from the worker on the local machine. I've seen examples of doing this in a sub-flow on a new deployment, which would be nice because you'd see failures in the main flow's status. But I can't find those examples right now. Anyway, we already have a utility method that schedules flows onto a different deployment in the future. In the simple case, we could reuse that code and schedule is for "now": https://github.com/als-computing/splash_flows_globus/blob/eec5c995b99b296e46cc323065339b09640cc9f5/orchestration/flows/bl832/move.py#L139
Talking with @davidalexandershapiro, he gave some interesting requirements for his flows:
Whenever a file is created, the file (in parrallel):
When the file is finished copying to NERSC
The low-res reconstruction lets the user quickly get some feedback on the success of the scan. The high-res will be used for actual research.
This introduces a few new concepts for his flows.
The main flow will invoke another flow, but not as a sub-flow. It will instead launch a new flow from a deployment so that it can get picked up from the worker on the local machine. I've seen examples of doing this in a sub-flow on a new deployment, which would be nice because you'd see failures in the main flow's status. But I can't find those examples right now. Anyway, we already have a utility method that schedules flows onto a different deployment in the future. In the simple case, we could reuse that code and schedule is for "now": https://github.com/als-computing/splash_flows_globus/blob/eec5c995b99b296e46cc323065339b09640cc9f5/orchestration/flows/bl832/move.py#L139