Push Data to GCS and BigQuery - Githubissues

coderxio / sagerx

Open drug data pipelines curated by pharmacists.

https://coderx.io/sagerx

Other

45 stars 12 forks source link

Push Data to GCS and BigQuery #299

Closed lprzychodzien closed 3 months ago

lprzychodzien commented 3 months ago

Explanation

Added functionality to upload to Google Cloud Storage (GCS) bucket
Added functionality to create datasets on Google Big Query (BQ)
Created a new postgres table to track available data materialized as tables (such as DBT) to then be able to reference for upload
Ran DBT runs in dbt container, using docker network
Created function to upload CSVs (sources) created by DAG to GCS

Tests

What testing did you do?
- Ran several DAGs
- Uploaded data to GCS and then linked to datasets on GCP though new DAG

jrlegrand commented 2 months ago

Fixes #295

jrlegrand commented 2 months ago

Fixes #287