GoogleCloudPlatform / DataflowTemplates

Cloud Dataflow Google-provided templates for solving in-Cloud data tasks
https://cloud.google.com/dataflow/docs/guides/templates/provided-templates
Apache License 2.0
1.16k stars 978 forks source link

Added Spanner Migration Tool alerting and monitoring code for sharded migrations #2017

Open nehamodgil opened 1 week ago

nehamodgil commented 1 week ago

This Terraform module delivers the following features tailored for the Spanner Migration Tool used in sharded migrations:

  1. Creates a custom Google Cloud monitoring dashboard to visualize critical metrics associated with the Spanner Migration Tool, enabling better monitoring and analysis.
  2. Configures alert policies to track and respond to errors and performance concerns during the migration process.
  3. Sets up notification channels to route alerts to designated endpoints, such as email or SMS
manitgupta commented 1 week ago

Sample Status check run - https://github.com/GoogleCloudPlatform/DataflowTemplates/actions/runs/11867244367/job/33121707907?pr=2017

This shows that the file is not formatted with terraform fmt.

codecov[bot] commented 1 week ago

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 52.93%. Comparing base (7fed825) to head (fe4a4d4). Report is 5 commits behind head on main.

Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #2017 +/- ## ============================================ + Coverage 45.41% 52.93% +7.51% + Complexity 3675 1367 -2308 ============================================ Files 842 378 -464 Lines 49970 20661 -29309 Branches 5261 2090 -3171 ============================================ - Hits 22692 10936 -11756 + Misses 25608 9045 -16563 + Partials 1670 680 -990 ``` | [Components](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/2017/components?src=pr&el=components&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform) | Coverage Δ | | |---|---|---| | [spanner-templates](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/2017/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform) | `67.93% <ø> (+1.22%)` | :arrow_up: | | [spanner-import-export](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/2017/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform) | `∅ <ø> (∅)` | | | [spanner-live-forward-migration](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/2017/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform) | `75.88% <ø> (ø)` | | | [spanner-live-reverse-replication](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/2017/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform) | `76.65% <ø> (ø)` | | | [spanner-bulk-migration](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/2017/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform) | `86.37% <ø> (ø)` | | [see 481 files with indirect coverage changes](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/2017/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform)