Set Up Airflow in Docker with ETL Tasks - Githubissues

sfbrigade / datasci-earthquake

MIT License

0 stars 3 forks source link

Set Up Airflow in Docker with ETL Tasks #74

Open agennadi opened 1 week ago

agennadi commented 1 week ago

Blocked by #73

Context

Set up an Airflow environment within Docker.
Define DAG for fetching and updating the data in DB tables - reuse ETL code from #73 for fetching the data from the external APIs.
Add tasks that check if there are updates in the external datasets.

Definition of Done

Airflow is containerized and running with Docker.
ETL workflows are operational as Airflow tasks.
Tests for the DAG are added.
Clear documentation on setting up and maintaining Airflow ETL tasks.

Engineering Details

Official Airflow in Docker documetation: https://airflow.apache.org/docs/apache-airflow/stable/howto/docker-compose/index.html
Simplified youtube turorial: https://www.youtube.com/watch?v=K9AnJ9_ZAXE&t=922s

Airflow Basic Pipeline

mackcooper1408 commented 1 week ago

this miiiiiight be blocked by getting a hosted PostgreSQL instance up and running as well... hopefully we'll get that up and running soon (next week or so)