Convert Workflows APIs ingestion logic into modular Airflow DAGs

smohiudd commented 7 months ago

What

The existing Workflows API includes ingestion logic in the API itself. To be able to generalize the Workflows API for different ingestion processes and stakeholders (i.e. MAAP and VEDA) the specific ingestions logic can be converted to modular Airflow DAGs. This flexibility will allow further user friendly ingestion workflows int the future.

PI Objective

Objective DS-3: Ingestion and Backend Improvements NASA-IMPACT/veda-architecture#455

Success Criteria

[x] Workflow API is converted to modular DAGs and deployed to veda-data-airflow dev

ividito commented 7 months ago

Met with @emileten to discuss this work today, using this rough diagram for how the new DAG should be organized.

Main ACs

Separation of dataset input into items and collections inputs, which go to disparate DAGs (this is already done in the Workflows API - we just need to move the logic.)
Orchestration so that item ingests will not be submitted until a corresponding collection (if present) has been ingested
Modular components - other DAGs should be able to reuse the "item discovery" and "collection ingestion sub-DAGs without changing their logic

Stretch AC

Extract summaries from built STAC items - make the collection DAG wait to incorporate these into the collection metadata prior to ingestion. This isn't needed to complete the task, but would be a nice improvement if it's easy to add during this refactor.

smohiudd commented 4 months ago

Airflow PR: https://github.com/NASA-IMPACT/veda-data-airflow/pull/159

NASA-IMPACT / veda-data-airflow