fix: upgrade dataplex tables to managed, create new zone, remove manual table creation

GoogleCloudPlatform / terraform-google-analytics-lakehouse

Deploys a Lakehouse Architecture Solution

Apache License 2.0

33 stars 16 forks source link

Closed bradmiro closed 1 year ago

bradmiro commented 1 year ago

This PR covers several changes to the codebase:

Separating the data into three dataplex zones: one raw and two curated
Creating a second staging bucket to separate image (raw) data from table (curated) data
Upgrading Dataplex assets to managed, programmatically (This feature is currently absent from the Terraform provider. FIled a FR with the eng team)
Parallelize the object copy step
Reorder Terraform graph to run Workflow after provisioning Dataplex