Automate scheduling and monitoring of workflow execution on CAVATICA

[ByroneCole-SageBionetworks]

• [ ] Follow the CHOP BixOps by processing the WGS and RNASeq on a subset of existing datasets (Khor /HTP) so we can confirm the expected output. The motivation for this is to:

• Learn how to manually trigger in case automation doesn’t work

• Learn the components required for automation

• Learn the CHOP BixOps

• [ ] Leverage Orca (Sage Workflow Orchestration) to launch, and monitor CHOP workflow for WGS and RNASeq processing (technology - Airflow) for open datasets and compare outputs against what’s expected.

• From a high level perspective, the automation would be done via an Airflow DAG. For simplicity, the high level steps in the Airflow DAG include:

  1. launch workflow(s) in Cavatica using SBG API
  2. monitor workflow(s) using SBP API + Airflow API
  3. notify when workflow(s) complete

• Orca is the combination of using Airflow + the Airflow DAG + the code that is required to launch workflows in Cavatica within the Airflow DAG. Explore if it is possible to convert CHOP BixOps into a DAG and make sure the final output from the Airflow DAG processing is the same as following CHOP BixOps “manually”

• [ ] Leverage Orca to launch, monitor CHOP workflows for WGS and RNASeq processing for new data.

[thomasyu888]

@ByroneCole-SageBionetworks Is this a duplicate of #29

[thomasyu888]

Closing this because this is a duplicated ticket