Utilizing the streamflow PBS connector should allow CWL based workflows produced by the MAAP to be broken up into 3 separate PBS jobs, stage-in, process, stage-out. Using streamflow may also be desirable from the perspective that workflows will also target kubernentes or AWS batch and streamflow workflows may be constructed in similar ways, therefore reducing the number of differences between the 3 different back-end processing systems. We investigate the possiblity and track here.
Definition of Done:
[x] Deliverable 1 - initial assessment of streamflow PBS connector on HECC
[x] Live Demo
[ ] Determine whether streamflow is required for R3 implementation
[ ] Deliverable 2 - prototype connector specific to HECC
[ ] Deliverable 3 - integrate prototype connector into ADES-PBS
Utilizing the streamflow PBS connector should allow CWL based workflows produced by the MAAP to be broken up into 3 separate PBS jobs, stage-in, process, stage-out. Using streamflow may also be desirable from the perspective that workflows will also target kubernentes or AWS batch and streamflow workflows may be constructed in similar ways, therefore reducing the number of differences between the 3 different back-end processing systems. We investigate the possiblity and track here.
Definition of Done: