search

AtlasOfLivingAustralia / extended-data-model

0 stars 0 forks source link

Consolidate occurrence and event airflow DAGs #79

Open djtfmartin opened 1 year ago

djtfmartin commented 1 year ago

We need to consolidate airflow DAGs for occurrence and event DwCA ingestion.

djtfmartin commented 1 year ago

Current airflow ingest DAG runs the following pipelines:

Screen Shot 2022-11-01 at 6 23 22 am

This can be enhanced to always run the DwCA export at the end:

Screen Shot 2022-11-01 at 6 23 29 am

And this is what it can look like with the additional events pipelines that are only ran for event based archives:

Screen Shot 2022-11-01 at 6 23 35 am
djtfmartin commented 1 year ago

Work ongoing on this branch: https://github.com/atlasoflivingaustralia/pipelines-airflow/tree/7_variables

djtfmartin commented 1 year ago

PR awaiting review https://github.com/AtlasOfLivingAustralia/pipelines-airflow/pull/14

javier-molina commented 1 year ago

To be reviewed after Airflow migration release.

This contains many changes that need to be integrated carefully