search

GoogleCloudPlatform / DataflowTemplates

Cloud Dataflow Google-provided templates for solving in-Cloud data tasks
https://cloud.google.com/dataflow/docs/guides/templates/provided-templates
Apache License 2.0
1.11k stars 931 forks source link

Live migration pre-configured profiles terraform sample #1668

Closed manitgupta closed 3 weeks ago

manitgupta commented 3 weeks ago

This PR adds samples for common scenarios users might have while trying to run a live migration to Spanner.

Each sample contains the following (and potentially more) files -

  1. main.tf - This contains the Terraform resources which will be created.
  2. outputs.tf - This declares the outputs that will be output as part of running the terraform example.
  3. variables.tf - This declares the input variables that are required to configure the resources.
  4. terraform.tf - This contains the required providers and APIs/project configurations for the sample.
  5. terraform.tfvars - This contains the dummy inputs that need to be populated to run the example.
  6. terraform_simple.tfvars - This contains the minimal list of dummy inputs that need to be populated to run the example.

SCENARIO: This Terraform example illustrates launching a live migration job for a MySQL source, given pre-created Datastream source and target connection profiles. As a result, it does not create any new buckets in the GCP account.

Creates the following resources -

  1. Pubsub topic and subscription - This contains GCS object notifications as files are written to GCS for consumption by the Dataflow job.
  2. Datastream stream - A datastream stream which reads from the source specified in the source connection profile and writes the data to the bucket specified in the target connection profile. Note that it uses a mandatory prefix path inside the bucket where it will write the data to. The default prefix path is data (can be overridden).
  3. Bucket notification - Creates the GCS bucket notification which publish to the pubsub topic created. Note that the bucket notification is created on the mandatory prefix path specified for the stream above.
  4. Dataflow job - The Dataflow job which reads from GCS and writes to Spanner.
  5. Permissions - It adds the required roles to the specified (or the default) service accounts for the live migration to work.
codecov[bot] commented 3 weeks ago

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 45.48%. Comparing base (66f4322) to head (e256f1b).

Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #1668 +/- ## ============================================ + Coverage 41.23% 45.48% +4.24% + Complexity 2928 717 -2211 ============================================ Files 763 301 -462 Lines 44393 16189 -28204 Branches 4754 1609 -3145 ============================================ - Hits 18305 7363 -10942 + Misses 24545 8284 -16261 + Partials 1543 542 -1001 ``` | [Components](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1668/components?src=pr&el=components&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform) | Coverage Δ | | |---|---|---| | [spanner-templates](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1668/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform) | `59.05% <ø> (-2.33%)` | :arrow_down: | | [spanner-import-export](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1668/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform) | `∅ <ø> (∅)` | | | [spanner-live-forward-migration](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1668/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform) | `73.88% <ø> (ø)` | | | [spanner-live-reverse-replication](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1668/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform) | `49.66% <ø> (ø)` | | | [spanner-bulk-migration](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1668/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform) | `82.07% <ø> (ø)` | | [see 478 files with indirect coverage changes](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1668/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform)