An organisation data package must be built as part of the Data Collection Pipeline. A DAG for this job must be published via the airflow-dags repository.
An ECS Task Definition and corresponding IAM execution role has already been provisioned via the airflow module in our Terraform provisioning code.
Create new python DAG generator file within airflow-dags under the dags directory
Include the organisation-builder within the trigger DAGs
For the scheduled trigger DAG, conditionally include organisation-builder based on whether the organisation-collection has been selected for scheduling
Acceptance Criteria/Tests
New DAG for organisation-builder running successfully within dev, staging and prod environments
New DAG runs after organisation-collection
Existing collection DAGs now run after organisation-builder
Resourcing & Dependencies
Are there any tickets that need to be completed before this one can be?
Yes, the data-package-builder needs to be checked to determine whether the code is up to date and ready for running within Airflow.
Are there any limitations as to who in the team can complete this ticket?
Access to Airflow needed and likely to need DevOps input
Are there any dependencies on other teams to assist, approve or be handed over to once deployed?
Overview
An organisation data package must be built as part of the Data Collection Pipeline. A DAG for this job must be published via the airflow-dags repository.
An ECS Task Definition and corresponding IAM execution role has already been provisioned via the airflow module in our Terraform provisioning code.
Tech Approach
Acceptance Criteria/Tests
Resourcing & Dependencies
Are there any tickets that need to be completed before this one can be?
Yes, the data-package-builder needs to be checked to determine whether the code is up to date and ready for running within Airflow.
Are there any limitations as to who in the team can complete this ticket?
Access to Airflow needed and likely to need DevOps input
Are there any dependencies on other teams to assist, approve or be handed over to once deployed?
None known.