search

GoogleCloudPlatform / DataflowTemplates

Cloud Dataflow Google-provided templates for solving in-Cloud data tasks
https://cloud.google.com/dataflow/docs/guides/templates/provided-templates
Apache License 2.0
1.12k stars 936 forks source link

Add CloudSql resource manager #1660

Closed Polber closed 1 month ago

Polber commented 1 month ago

PR adds the CloudSqlResourceManager to manage SQL resources on GCP.

For MySQL - expects a static CloudSQL instance running MySQL For Oracle - expects a static GCE VM running Oracle XE

codecov[bot] commented 1 month ago

Codecov Report

Attention: Patch coverage is 82.50000% with 21 lines in your changes missing coverage. Please review.

Project coverage is 41.28%. Comparing base (f1a52d0) to head (5838d4e). Report is 29 commits behind head on main.

Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #1660 +/- ## ============================================ - Coverage 45.53% 41.28% -4.25% - Complexity 717 2929 +2212 ============================================ Files 301 769 +468 Lines 16171 44568 +28397 Branches 1609 4768 +3159 ============================================ + Hits 7363 18399 +11036 - Misses 8266 24621 +16355 - Partials 542 1548 +1006 ``` | [Components](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1660/components?src=pr&el=components&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform) | Coverage Δ | | |---|---|---| | [spanner-templates](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1660/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform) | `61.40% <ø> (+2.34%)` | :arrow_up: | | [spanner-import-export](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1660/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform) | `64.44% <ø> (∅)` | | | [spanner-live-forward-migration](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1660/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform) | `73.87% <ø> (-0.01%)` | :arrow_down: | | [spanner-live-reverse-replication](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1660/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform) | `49.70% <ø> (+0.04%)` | :arrow_up: | | [spanner-bulk-migration](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1660/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform) | `82.06% <ø> (-0.01%)` | :arrow_down: | | [Files](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1660?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform) | Coverage Δ | | |---|---|---| | [.../it/gcp/bigquery/BigQueryResourceManagerUtils.java](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1660?src=pr&el=tree&filepath=it%2Fgoogle-cloud-platform%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fbeam%2Fit%2Fgcp%2Fbigquery%2FBigQueryResourceManagerUtils.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform#diff-aXQvZ29vZ2xlLWNsb3VkLXBsYXRmb3JtL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9iZWFtL2l0L2djcC9iaWdxdWVyeS9CaWdRdWVyeVJlc291cmNlTWFuYWdlclV0aWxzLmphdmE=) | `83.33% <ø> (ø)` | | | [.../it/gcp/bigtable/BigtableResourceManagerUtils.java](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1660?src=pr&el=tree&filepath=it%2Fgoogle-cloud-platform%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fbeam%2Fit%2Fgcp%2Fbigtable%2FBigtableResourceManagerUtils.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform#diff-aXQvZ29vZ2xlLWNsb3VkLXBsYXRmb3JtL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9iZWFtL2l0L2djcC9iaWd0YWJsZS9CaWd0YWJsZVJlc291cmNlTWFuYWdlclV0aWxzLmphdmE=) | `76.92% <ø> (ø)` | | | [...eam/it/gcp/cloudsql/CloudMySQLResourceManager.java](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1660?src=pr&el=tree&filepath=it%2Fgoogle-cloud-platform%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fbeam%2Fit%2Fgcp%2Fcloudsql%2FCloudMySQLResourceManager.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform#diff-aXQvZ29vZ2xlLWNsb3VkLXBsYXRmb3JtL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9iZWFtL2l0L2djcC9jbG91ZHNxbC9DbG91ZE15U1FMUmVzb3VyY2VNYW5hZ2VyLmphdmE=) | `100.00% <100.00%> (ø)` | | | [...gcp/datastream/DatastreamResourceManagerUtils.java](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1660?src=pr&el=tree&filepath=it%2Fgoogle-cloud-platform%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fbeam%2Fit%2Fgcp%2Fdatastream%2FDatastreamResourceManagerUtils.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform#diff-aXQvZ29vZ2xlLWNsb3VkLXBsYXRmb3JtL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9iZWFtL2l0L2djcC9kYXRhc3RyZWFtL0RhdGFzdHJlYW1SZXNvdXJjZU1hbmFnZXJVdGlscy5qYXZh) | `100.00% <ø> (ø)` | | | [...gcp/spanner/utils/SpannerResourceManagerUtils.java](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1660?src=pr&el=tree&filepath=it%2Fgoogle-cloud-platform%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fbeam%2Fit%2Fgcp%2Fspanner%2Futils%2FSpannerResourceManagerUtils.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform#diff-aXQvZ29vZ2xlLWNsb3VkLXBsYXRmb3JtL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9iZWFtL2l0L2djcC9zcGFubmVyL3V0aWxzL1NwYW5uZXJSZXNvdXJjZU1hbmFnZXJVdGlscy5qYXZh) | `97.50% <ø> (ø)` | | | [.../it/gcp/cloudsql/CloudSqlResourceManagerUtils.java](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1660?src=pr&el=tree&filepath=it%2Fgoogle-cloud-platform%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fbeam%2Fit%2Fgcp%2Fcloudsql%2FCloudSqlResourceManagerUtils.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform#diff-aXQvZ29vZ2xlLWNsb3VkLXBsYXRmb3JtL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9iZWFtL2l0L2djcC9jbG91ZHNxbC9DbG91ZFNxbFJlc291cmNlTWFuYWdlclV0aWxzLmphdmE=) | `92.30% <92.30%> (ø)` | | | [...am/it/gcp/cloudsql/CloudOracleResourceManager.java](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1660?src=pr&el=tree&filepath=it%2Fgoogle-cloud-platform%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fbeam%2Fit%2Fgcp%2Fcloudsql%2FCloudOracleResourceManager.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform#diff-aXQvZ29vZ2xlLWNsb3VkLXBsYXRmb3JtL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9iZWFtL2l0L2djcC9jbG91ZHNxbC9DbG91ZE9yYWNsZVJlc291cmNlTWFuYWdlci5qYXZh) | `85.00% <85.00%> (ø)` | | | [...apache/beam/it/gcp/cloudsql/CloudSqlContainer.java](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1660?src=pr&el=tree&filepath=it%2Fgoogle-cloud-platform%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fbeam%2Fit%2Fgcp%2Fcloudsql%2FCloudSqlContainer.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform#diff-aXQvZ29vZ2xlLWNsb3VkLXBsYXRmb3JtL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9iZWFtL2l0L2djcC9jbG91ZHNxbC9DbG91ZFNxbENvbnRhaW5lci5qYXZh) | `80.00% <80.00%> (ø)` | | | [.../beam/it/gcp/cloudsql/CloudSqlResourceManager.java](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1660?src=pr&el=tree&filepath=it%2Fgoogle-cloud-platform%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fbeam%2Fit%2Fgcp%2Fcloudsql%2FCloudSqlResourceManager.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform#diff-aXQvZ29vZ2xlLWNsb3VkLXBsYXRmb3JtL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9iZWFtL2l0L2djcC9jbG91ZHNxbC9DbG91ZFNxbFJlc291cmNlTWFuYWdlci5qYXZh) | `78.46% <78.46%> (ø)` | | ... and [484 files with indirect coverage changes](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1660/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform)
Abacn commented 1 month ago

The Dataflow Integration Test failed on

JdbcToBigQueryIT.testMySqlToBigQueryWithStorageWriteApi

looks relevant

Polber commented 1 month ago

The Dataflow Integration Test failed on

JdbcToBigQueryIT.testMySqlToBigQueryWithStorageWriteApi

looks relevant

Shouldn't be relevant since that test does not use CloudSqlResourceManager, but running again just to be safe

Abacn commented 1 month ago

This time is MqttToPubsubTestIT.testMqttToPubSub failed with HTTP 429. Different tests flaky.

Abacn commented 1 month ago 
"message": "Quota exceeded for quota metric 'Job update requests' and limit 'Job update requests per minute per user' of service 'dataflow.googleapis.com' for consumer 'project_number:269744978479'.",

we may want to address this in integration test framework later