Airflow Test Deployment

Deploy an Airflow instance with CWL support to the unity environment.
Run a chirp rebin worfklow on a multi-node airflow cluster
Document task stage-in/out complexity *
Investigate hybrid setups (on-prem / AWS airflow workers)
- create future work issues.features for the hybrid setup
Manually deployed cwl workflows to the system e.g. not through WPS-T
Expose Airflow UI through API Gateway (not directly!)
- Add cognito authentication to Airflow UI ?? Or rely on lambda/auth at the API Gateway level?
Execution of workflow through Airflow UI

Stretch: Add WPS-T front end for process execution and monitoring

I fear that stage-in/stage-out will be impacted by the move to airflow. IT might be minimal (that is, we still run stage-in/out but ensure they are run on the same node as the 'process' step. Or it might be much larger (Task pre/post executors). The reason i have this concern is that each CWL step is a separate 'task' in ariflow, which means stage-in, process, and stage-out might all happen on different nodes. I'd rather not try and use EFS for this (slow) but we might not have another option.

unity-sds / unity-project-management