[ ] Follow the CHOP BixOps by processing the WGS and RNASeq on a subset of existing datasets (Khor /HTP) so we can confirm the expected output. The motivation for this is to:
Learn how to manually trigger in case automation doesn’t work
Learn the components required for automation
Learn the CHOP BixOps
[ ] Leverage Orca (Sage Workflow Orchestration) to launch, and monitor CHOP workflow for WGS and RNASeq processing (technology - Airflow) for open datasets and compare outputs against what’s expected.
From a high level perspective, the automation would be done via an Airflow DAG. For simplicity, the high level steps in the Airflow DAG include:
launch workflow(s) in Cavatica using SBG API
monitor workflow(s) using SBP API + Airflow API
notify when workflow(s) complete
Orca is the combination of using Airflow + the Airflow DAG + the code that is required to launch workflows in Cavatica within the Airflow DAG. Explore if it is possible to convert CHOP BixOps into a DAG and make sure the final output from the Airflow DAG processing is the same as following CHOP BixOps “manually”
[ ] Leverage Orca to launch, monitor CHOP workflows for WGS and RNASeq processing for new data.
[ ] Follow the CHOP BixOps by processing the WGS and RNASeq on a subset of existing datasets (Khor /HTP) so we can confirm the expected output. The motivation for this is to:
Learn how to manually trigger in case automation doesn’t work
Learn the components required for automation
Learn the CHOP BixOps
[ ] Leverage Orca (Sage Workflow Orchestration) to launch, and monitor CHOP workflow for WGS and RNASeq processing (technology - Airflow) for open datasets and compare outputs against what’s expected.
From a high level perspective, the automation would be done via an Airflow DAG. For simplicity, the high level steps in the Airflow DAG include:
Orca is the combination of using Airflow + the Airflow DAG + the code that is required to launch workflows in Cavatica within the Airflow DAG. Explore if it is possible to convert CHOP BixOps into a DAG and make sure the final output from the Airflow DAG processing is the same as following CHOP BixOps “manually”
[ ] Leverage Orca to launch, monitor CHOP workflows for WGS and RNASeq processing for new data.