Some changes:
Composer Version Upgrade - Upgraded Composer from composer-2.0.31-airflow-2.3.3 to composer-2.2.0-airflow-2.5.1
Maintenance - Moved Composer, Cloud SQL, Datastream and Spanner to us-west2 region (from us-central1)
Datastream - When destroying datastream the airflow job will delete the CDC marker file so the data restarts at “zero”.
Cloud NAT - Get a distinct list of regions in the deployment and dynamically create a NAT per region. Since the system uses private IP address you need a NAT per region to gain internet access from VMs.
Correction - Fix instructions for Rideshare Lakehouse website for linking Looker dashboard
Some changes: Composer Version Upgrade - Upgraded Composer from composer-2.0.31-airflow-2.3.3 to composer-2.2.0-airflow-2.5.1 Maintenance - Moved Composer, Cloud SQL, Datastream and Spanner to us-west2 region (from us-central1) Datastream - When destroying datastream the airflow job will delete the CDC marker file so the data restarts at “zero”. Cloud NAT - Get a distinct list of regions in the deployment and dynamically create a NAT per region. Since the system uses private IP address you need a NAT per region to gain internet access from VMs. Correction - Fix instructions for Rideshare Lakehouse website for linking Looker dashboard