search

GoogleCloudPlatform / DataflowTemplates

Cloud Dataflow Google-provided templates for solving in-Cloud data tasks
https://cloud.google.com/dataflow/docs/guides/templates/provided-templates
Apache License 2.0
1.15k stars 967 forks source link

pipeline parameter updates #1740

Closed bharadwaj-aditya closed 2 months ago

bharadwaj-aditya commented 3 months ago

Changes to the parameters

  1. Table seperator changed to pipe and comma - to support gcloud handling
  2. removed hard checks on session file mapping
  3. Converted RPC priority to high
  4. Rename sourceDB url to sourceConfig url
codecov[bot] commented 3 months ago

Codecov Report

Attention: Patch coverage is 82.60870% with 4 lines in your changes missing coverage. Please review.

Project coverage is 49.42%. Comparing base (8fb7b89) to head (571c2c4). Report is 3 commits behind head on main.

Files Patch % Lines
...loud/teleport/v2/templates/PipelineController.java 33.33% 2 Missing :warning:
...cloud/teleport/v2/templates/SourceDbToSpanner.java 0.00% 2 Missing :warning:
Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #1740 +/- ## ============================================ + Coverage 42.58% 49.42% +6.84% + Complexity 3365 1226 -2139 ============================================ Files 815 351 -464 Lines 47677 19258 -28419 Branches 5109 1941 -3168 ============================================ - Hits 20304 9519 -10785 + Misses 25717 9085 -16632 + Partials 1656 654 -1002 ``` | [Components](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1740/components?src=pr&el=components&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform) | Coverage Δ | | |---|---|---| | [spanner-templates](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1740/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform) | `64.84% <82.60%> (+0.27%)` | :arrow_up: | | [spanner-import-export](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1740/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform) | `∅ <ø> (∅)` | | | [spanner-live-forward-migration](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1740/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform) | `75.37% <100.00%> (+0.37%)` | :arrow_up: | | [spanner-live-reverse-replication](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1740/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform) | `52.28% <100.00%> (+0.31%)` | :arrow_up: | | [spanner-bulk-migration](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1740/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform) | `83.69% <82.60%> (+0.22%)` | :arrow_up: | | [Files](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1740?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform) | Coverage Δ | | |---|---|---| | [...ud/teleport/v2/options/OptionsToConfigBuilder.java](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1740?src=pr&el=tree&filepath=v2%2Fsourcedb-to-spanner%2Fsrc%2Fmain%2Fjava%2Fcom%2Fgoogle%2Fcloud%2Fteleport%2Fv2%2Foptions%2FOptionsToConfigBuilder.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform#diff-djIvc291cmNlZGItdG8tc3Bhbm5lci9zcmMvbWFpbi9qYXZhL2NvbS9nb29nbGUvY2xvdWQvdGVsZXBvcnQvdjIvb3B0aW9ucy9PcHRpb25zVG9Db25maWdCdWlsZGVyLmphdmE=) | `87.14% <100.00%> (+0.37%)` | :arrow_up: | | [.../spanner/migrations/schema/SessionBasedMapper.java](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1740?src=pr&el=tree&filepath=v2%2Fspanner-common%2Fsrc%2Fmain%2Fjava%2Fcom%2Fgoogle%2Fcloud%2Fteleport%2Fv2%2Fspanner%2Fmigrations%2Fschema%2FSessionBasedMapper.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform#diff-djIvc3Bhbm5lci1jb21tb24vc3JjL21haW4vamF2YS9jb20vZ29vZ2xlL2Nsb3VkL3RlbGVwb3J0L3YyL3NwYW5uZXIvbWlncmF0aW9ucy9zY2hlbWEvU2Vzc2lvbkJhc2VkTWFwcGVyLmphdmE=) | `100.00% <100.00%> (ø)` | | | [...loud/teleport/v2/templates/PipelineController.java](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1740?src=pr&el=tree&filepath=v2%2Fsourcedb-to-spanner%2Fsrc%2Fmain%2Fjava%2Fcom%2Fgoogle%2Fcloud%2Fteleport%2Fv2%2Ftemplates%2FPipelineController.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform#diff-djIvc291cmNlZGItdG8tc3Bhbm5lci9zcmMvbWFpbi9qYXZhL2NvbS9nb29nbGUvY2xvdWQvdGVsZXBvcnQvdjIvdGVtcGxhdGVzL1BpcGVsaW5lQ29udHJvbGxlci5qYXZh) | `20.27% <33.33%> (-0.15%)` | :arrow_down: | | [...cloud/teleport/v2/templates/SourceDbToSpanner.java](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1740?src=pr&el=tree&filepath=v2%2Fsourcedb-to-spanner%2Fsrc%2Fmain%2Fjava%2Fcom%2Fgoogle%2Fcloud%2Fteleport%2Fv2%2Ftemplates%2FSourceDbToSpanner.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform#diff-djIvc291cmNlZGItdG8tc3Bhbm5lci9zcmMvbWFpbi9qYXZhL2NvbS9nb29nbGUvY2xvdWQvdGVsZXBvcnQvdjIvdGVtcGxhdGVzL1NvdXJjZURiVG9TcGFubmVyLmphdmE=) | `31.57% <0.00%> (ø)` | | ... and [489 files with indirect coverage changes](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1740/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform)