GoogleCloudPlatform / DataflowTemplates

Cloud Dataflow Google-provided templates for solving in-Cloud data tasks
https://cloud.google.com/dataflow/docs/guides/templates/provided-templates
Apache License 2.0
1.14k stars 960 forks source link

Terraform Sample: MySQL database setup for private connectivity #1672

Closed manitgupta closed 3 months ago

manitgupta commented 3 months ago

This Terraform example illustrates launching a MySQL 5.7 in a GCE Compute instance inside a custom VPC subnet. It adds firewall rules to ensure that 1) Datastream can connect to the MySQL via private connectivity and 2) Dataflow VMs can communicate with each other. It also creates a Spanner instance and database to migrate data to.

codecov[bot] commented 3 months ago

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 45.45%. Comparing base (f14c62c) to head (30a5a21). Report is 46 commits behind head on main.

:exclamation: Current head 30a5a21 differs from pull request most recent head 8543bce

Please upload reports for the commit 8543bce to get more accurate results.

Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #1672 +/- ## ============================================ + Coverage 41.29% 45.45% +4.16% + Complexity 2929 717 -2212 ============================================ Files 769 301 -468 Lines 44602 16181 -28421 Branches 4770 1607 -3163 ============================================ - Hits 18418 7355 -11063 + Misses 24636 8286 -16350 + Partials 1548 540 -1008 ``` | [Components](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1672/components?src=pr&el=components&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform) | Coverage Δ | | |---|---|---| | [spanner-templates](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1672/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform) | `59.10% <ø> (-2.31%)` | :arrow_down: | | [spanner-import-export](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1672/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform) | `∅ <ø> (∅)` | | | [spanner-live-forward-migration](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1672/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform) | `73.87% <ø> (ø)` | | | [spanner-live-reverse-replication](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1672/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform) | `49.70% <ø> (ø)` | | | [spanner-bulk-migration](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1672/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform) | `82.06% <ø> (ø)` | | [see 491 files with indirect coverage changes](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1672/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform)
manitgupta commented 3 months ago

Sure, I am removing the Reverse replication links from here, and adding a TODO here to add these links to the future. I think we have been rehashed the same thing a number of times in the past. I think some strong words are used every time this topic is brought up so I am starting a thread internally to close on this topic.

aksharauke commented 3 months ago

Sure, I am removing the Reverse replication links from here, and adding a TODO here to add these links to the future. I think we have been rehashed the same thing a number of times in the past. I think some strong words are used every time this topic is brought up so I am starting a thread internally to close on this topic.

Sure. I am not privy to any past discussion where I consented to allowing user to figure out reverse replication just based on Dataflow readmes. Happy to discuss internally.

manitgupta commented 3 months ago

I am not privy to any past discussion where I consented to allowing user to figure out reverse replication just based on Dataflow readmes

I don't think this about this at all. The documentation reads -

These are especially useful is setting up test infrastructure that resembles a customers' environment, in order to run various Spanner migration tooling such as - These are simply links to the migration products, rather than a prescription of how to use them.

aksharauke commented 3 months ago

I am not privy to any past discussion where I consented to allowing user to figure out reverse replication just based on Dataflow readmes

I don't think this about this at all. The documentation reads -

These are especially useful is setting up test infrastructure that resembles a customers' environment, in order to run various Spanner migration tooling such as - These are simply links to the migration products, rather than a prescription of how to use them.

I thought we wanted to discuss this internally. Happy to continue here as well. The description above itself refers to Spanner migration tooling, the Dataflow template alone is not a tool for reverse replication - there is more to it. Also, there are 2 templates - only one was linked, which was wrong also. So for reverse, some thought is needed before putting out the links - so that it's more intuitive for a first time user. Hence the request for adding a TODO.

manitgupta commented 3 months ago

Thanks for the link. I think it's a good idea! For now, I would like to keep these examples colocated with the Dataflow jobs for discoverability!