search

GoogleCloudPlatform / DataflowTemplates

Cloud Dataflow Google-provided templates for solving in-Cloud data tasks
https://cloud.google.com/dataflow/docs/guides/templates/provided-templates
Apache License 2.0
1.14k stars 950 forks source link

Add App Profile ID to CBT Dataflow Templates: Bigtable to Avro/Parquet/JSON #1762

Closed djyau closed 1 month ago

djyau commented 1 month ago

Add an optional app profile ID parameter to the following classic templates:

If not supplied, uses "default" app profile by default.

codecov[bot] commented 1 month ago

Codecov Report

Attention: Patch coverage is 0% with 4 lines in your changes missing coverage. Please review.

Project coverage is 42.86%. Comparing base (235a4a3) to head (17e6047). Report is 5 commits behind head on main.

Files Patch % Lines
.../it/gcp/bigtable/BigtableResourceManagerUtils.java 0.00% 1 Missing :warning:
...google/cloud/teleport/bigtable/BigtableToAvro.java 0.00% 1 Missing :warning:
...google/cloud/teleport/bigtable/BigtableToJson.java 0.00% 1 Missing :warning:
...gle/cloud/teleport/bigtable/BigtableToParquet.java 0.00% 1 Missing :warning:
Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #1762 +/- ## ============================================ + Coverage 42.57% 42.86% +0.29% - Complexity 3357 3398 +41 ============================================ Files 815 815 Lines 47672 47707 +35 Branches 5107 5110 +3 ============================================ + Hits 20294 20449 +155 + Misses 25718 25599 -119 + Partials 1660 1659 -1 ``` | [Components](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1762/components?src=pr&el=components&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform) | Coverage Δ | | |---|---|---| | [spanner-templates](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1762/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform) | `64.65% <ø> (+0.14%)` | :arrow_up: | | [spanner-import-export](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1762/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform) | `64.40% <ø> (+0.10%)` | :arrow_up: | | [spanner-live-forward-migration](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1762/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform) | `75.30% <ø> (+0.29%)` | :arrow_up: | | [spanner-live-reverse-replication](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1762/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform) | `52.17% <ø> (+0.21%)` | :arrow_up: | | [spanner-bulk-migration](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1762/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform) | `83.68% <ø> (+0.23%)` | :arrow_up: | | [Files](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1762?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform) | Coverage Δ | | |---|---|---| | [.../it/gcp/bigtable/BigtableResourceManagerUtils.java](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1762?src=pr&el=tree&filepath=it%2Fgoogle-cloud-platform%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fbeam%2Fit%2Fgcp%2Fbigtable%2FBigtableResourceManagerUtils.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform#diff-aXQvZ29vZ2xlLWNsb3VkLXBsYXRmb3JtL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9iZWFtL2l0L2djcC9iaWd0YWJsZS9CaWd0YWJsZVJlc291cmNlTWFuYWdlclV0aWxzLmphdmE=) | `75.47% <0.00%> (-1.46%)` | :arrow_down: | | [...google/cloud/teleport/bigtable/BigtableToAvro.java](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1762?src=pr&el=tree&filepath=v1%2Fsrc%2Fmain%2Fjava%2Fcom%2Fgoogle%2Fcloud%2Fteleport%2Fbigtable%2FBigtableToAvro.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform#diff-djEvc3JjL21haW4vamF2YS9jb20vZ29vZ2xlL2Nsb3VkL3RlbGVwb3J0L2JpZ3RhYmxlL0JpZ3RhYmxlVG9BdnJvLmphdmE=) | `35.71% <0.00%> (-0.65%)` | :arrow_down: | | [...google/cloud/teleport/bigtable/BigtableToJson.java](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1762?src=pr&el=tree&filepath=v1%2Fsrc%2Fmain%2Fjava%2Fcom%2Fgoogle%2Fcloud%2Fteleport%2Fbigtable%2FBigtableToJson.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform#diff-djEvc3JjL21haW4vamF2YS9jb20vZ29vZ2xlL2Nsb3VkL3RlbGVwb3J0L2JpZ3RhYmxlL0JpZ3RhYmxlVG9Kc29uLmphdmE=) | `62.63% <0.00%> (-0.70%)` | :arrow_down: | | [...gle/cloud/teleport/bigtable/BigtableToParquet.java](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1762?src=pr&el=tree&filepath=v1%2Fsrc%2Fmain%2Fjava%2Fcom%2Fgoogle%2Fcloud%2Fteleport%2Fbigtable%2FBigtableToParquet.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform#diff-djEvc3JjL21haW4vamF2YS9jb20vZ29vZ2xlL2Nsb3VkL3RlbGVwb3J0L2JpZ3RhYmxlL0JpZ3RhYmxlVG9QYXJxdWV0LmphdmE=) | `40.90% <0.00%> (-0.96%)` | :arrow_down: | ... and [14 files with indirect coverage changes](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1762/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform)