Automate ingest GCS to BQ data load

google / project-OCEAN

Project OCEAN is an open science collaboration focused on understanding the open source ecosystems creating datasets that enable research and forming a clear understanding of the state of open source communities.

https://vermontcomplexsystems.org/partner/OCEAN/

Apache License 2.0

50 stars 19 forks source link

Automate ingest GCS to BQ data load #26

Closed nyghtowl closed 3 years ago

nyghtowl commented 4 years ago

Add automation ingesting data from GCS to BQ

Proposal is use Cloud Functions with triggers on GCS buckets

nyghtowl commented 4 years ago

Setup Cloud Functions / Data Flow? triggered by GCS bucket updates to clean up and and load BQ
Evaluate if need to include with Run/Build