search

NASA-IMPACT / veda-data-airflow

Airflow implementation of ingest pipeline for VEDA STAC data
Other
7 stars 4 forks source link

ADR discussing status tracking #97

Closed moradology closed 9 months ago

moradology commented 1 year ago

It is not uncommon to see even far simpler ETL and async processing pipelines than what we're aiming to create with robust status-tracking, ingest-identification, and awareness/record keeping around the versions of all software used during ingest. It would be useful for us to consider what our options are to keep track of this kind of information. Perhaps a dynamo database that can track status and other relevant information? Maybe it could be used by the ingestor api as well to clarify system behavior to third parties

moradology commented 1 year ago

This is also likely to be useful for QA purposes and we should consider their needs when working this out

gadomski commented 1 year ago

Moving to https://github.com/NASA-IMPACT/veda-data-airflow/ as this is tagged airflow. Please feel free to close if this is OBE.