GoogleCloudPlatform / DataflowTemplates

Cloud Dataflow Google-provided templates for solving in-Cloud data tasks
https://cloud.google.com/dataflow/docs/guides/templates/provided-templates
Apache License 2.0
1.11k stars 931 forks source link

Terraform: Add support for using pre-existing private connectivity configuration #1692

Closed manitgupta closed 1 week ago

manitgupta commented 2 weeks ago
  1. Adds ability to specify a private connectivity configuration for the end-to-end infrastructure setup Terraform template. Private connectivity requires sensitive permissions hence the users might want to set it up manually instead of relying on automation.
  2. Adds ability to configure spanner_project_id for cross-project writes.

Note: I had spent considerable time setting up MySQL 8.4 VPC setup by self-discovering how to configure a MySQL self-hosted database to work with Datastream. While I made numerous changes as per the official docs of MySQL, I am still running into errors. Current Datastream instructions are for MySQL 5.7 - https://cloud.google.com/datastream/docs/configure-self-managed-mysql. I will ask Datastream to provide documentation on how to configure MySQL 8, in the meantime switching back to MySQL 5.7.