Set Up Airflow for Data Ingestion and Transformation

olgazju / data_analytics_dashboard_starter_kit

An easy-to-use Data Analytics Dashboard Starter Kit with FastAPI, Neon, Airflow, and Streamlit. Designed to help developers get started with Neon and quickly integrate, process, and visualize data for building analytics applications.

Apache License 2.0

1 stars 0 forks source link

Set Up Airflow for Data Ingestion and Transformation #2

Closed olgazju closed 2 months ago

olgazju commented 2 months ago

Set up Apache Airflow in a new Docker container using Docker Compose. Create an Airflow DAG (Directed Acyclic Graph) to simulate data extraction, transformation, and loading into the Neon database using the SQLAlchemy library or directly through a connection to Neon.

olgazju commented 2 months ago

Need to try https://cloud.astronomer.io/cm073qewl0ays01n5rnugks3a/dags

olgazju commented 2 months ago

using Astronomer