R&D: Re-think data ingestion infra - Githubissues

the-data-base / nhl-data

Repo for version control the dbt transformations on nhl-breakouts data

5 stars 0 forks source link

R&D: Re-think data ingestion infra #72

Open dmf95 opened 2 years ago

dmf95 commented 2 years ago

Brief

How is the ingestion of data into the dbt-nhl-breakouts project being triggered in a "better" state... still through GH actions? Prefect? Airflow?
Is there a way to trigger a stream without needing its dependency? De-coupling different streams?

AC

Plan...No need to click anything to get the data unless we needed a backfill
Plan... Moving away from GH actions towards something with more control (no UI needed, all code... more control over task env)
Plan... Keeping compute infra in-mind when designing this