breedfides / airflow-etl

0 stars 2 forks source link

Joint architecture concept with BreedFides Portal #4

Closed gannebamm closed 9 months ago

gannebamm commented 9 months ago

Status quo

We only have a very draft version of the architecture of how the BreedFidel Portal frontend will trigger AirFlow and what Payload has to be sent. Therefore, we shall schedule a meeting with both teams.

Expeceted behaviour

After the meeting with both teams, we have a joint concept and a well-defined JSON payload to trigger AirFlow DAGs.

gannebamm commented 9 months ago

The meeting was successful. We discussed using one primary DAG as entry point for the ETL pipeline. See: https://github.com/breedfides/airflow-etl/issues/7