GoogleCloudPlatform / DataflowTemplates

Cloud Dataflow Google-provided templates for solving in-Cloud data tasks
https://cloud.google.com/dataflow/docs/guides/templates/provided-templates
Apache License 2.0
1.12k stars 936 forks source link

[BULK] Shard id population in transformer #1633

Closed Deep1998 closed 1 month ago

codecov[bot] commented 1 month ago

Codecov Report

Attention: Patch coverage is 98.03922% with 1 line in your changes missing coverage. Please review.

Project coverage is 45.58%. Comparing base (8674def) to head (2d8da06). Report is 19 commits behind head on main.

Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #1633 +/- ## ============================================ + Coverage 41.49% 45.58% +4.08% + Complexity 2922 720 -2202 ============================================ Files 754 301 -453 Lines 43983 16192 -27791 Branches 4707 1611 -3096 ============================================ - Hits 18252 7381 -10871 + Misses 24193 8269 -15924 + Partials 1538 542 -996 ``` | [Components](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1633/components?src=pr&el=components&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform) | Coverage Δ | | |---|---|---| | [spanner-templates](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1633/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform) | `59.14% <98.03%> (-2.14%)` | :arrow_down: | | [spanner-import-export](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1633/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform) | `∅ <ø> (∅)` | | | [spanner-live-forward-migration](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1633/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform) | `73.88% <97.87%> (+0.31%)` | :arrow_up: | | [spanner-live-reverse-replication](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1633/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform) | `49.66% <97.87%> (+0.50%)` | :arrow_up: | | [spanner-bulk-migration](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1633/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform) | `82.09% <98.03%> (+0.17%)` | :arrow_up: | | [Files](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1633?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform) | Coverage Δ | | |---|---|---| | [.../reader/io/jdbc/rowmapper/JdbcSourceRowMapper.java](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1633?src=pr&el=tree&filepath=v2%2Fsourcedb-to-spanner%2Fsrc%2Fmain%2Fjava%2Fcom%2Fgoogle%2Fcloud%2Fteleport%2Fv2%2Fsource%2Freader%2Fio%2Fjdbc%2Frowmapper%2FJdbcSourceRowMapper.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform#diff-djIvc291cmNlZGItdG8tc3Bhbm5lci9zcmMvbWFpbi9qYXZhL2NvbS9nb29nbGUvY2xvdWQvdGVsZXBvcnQvdjIvc291cmNlL3JlYWRlci9pby9qZGJjL3Jvd21hcHBlci9KZGJjU291cmNlUm93TWFwcGVyLmphdmE=) | `100.00% <100.00%> (ø)` | | | [...ud/teleport/v2/source/reader/io/row/SourceRow.java](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1633?src=pr&el=tree&filepath=v2%2Fsourcedb-to-spanner%2Fsrc%2Fmain%2Fjava%2Fcom%2Fgoogle%2Fcloud%2Fteleport%2Fv2%2Fsource%2Freader%2Fio%2Frow%2FSourceRow.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform#diff-djIvc291cmNlZGItdG8tc3Bhbm5lci9zcmMvbWFpbi9qYXZhL2NvbS9nb29nbGUvY2xvdWQvdGVsZXBvcnQvdjIvc291cmNlL3JlYWRlci9pby9yb3cvU291cmNlUm93LmphdmE=) | `100.00% <100.00%> (ø)` | | | [...cloud/teleport/v2/templates/SourceDbToSpanner.java](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1633?src=pr&el=tree&filepath=v2%2Fsourcedb-to-spanner%2Fsrc%2Fmain%2Fjava%2Fcom%2Fgoogle%2Fcloud%2Fteleport%2Fv2%2Ftemplates%2FSourceDbToSpanner.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform#diff-djIvc291cmNlZGItdG8tc3Bhbm5lci9zcmMvbWFpbi9qYXZhL2NvbS9nb29nbGUvY2xvdWQvdGVsZXBvcnQvdjIvdGVtcGxhdGVzL1NvdXJjZURiVG9TcGFubmVyLmphdmE=) | `49.42% <ø> (ø)` | | | [...leport/v2/transformer/SourceRowToMutationDoFn.java](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1633?src=pr&el=tree&filepath=v2%2Fsourcedb-to-spanner%2Fsrc%2Fmain%2Fjava%2Fcom%2Fgoogle%2Fcloud%2Fteleport%2Fv2%2Ftransformer%2FSourceRowToMutationDoFn.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform#diff-djIvc291cmNlZGItdG8tc3Bhbm5lci9zcmMvbWFpbi9qYXZhL2NvbS9nb29nbGUvY2xvdWQvdGVsZXBvcnQvdjIvdHJhbnNmb3JtZXIvU291cmNlUm93VG9NdXRhdGlvbkRvRm4uamF2YQ==) | `97.29% <100.00%> (ø)` | | | [...t/v2/spanner/migrations/schema/IdentityMapper.java](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1633?src=pr&el=tree&filepath=v2%2Fspanner-common%2Fsrc%2Fmain%2Fjava%2Fcom%2Fgoogle%2Fcloud%2Fteleport%2Fv2%2Fspanner%2Fmigrations%2Fschema%2FIdentityMapper.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform#diff-djIvc3Bhbm5lci1jb21tb24vc3JjL21haW4vamF2YS9jb20vZ29vZ2xlL2Nsb3VkL3RlbGVwb3J0L3YyL3NwYW5uZXIvbWlncmF0aW9ucy9zY2hlbWEvSWRlbnRpdHlNYXBwZXIuamF2YQ==) | `100.00% <100.00%> (ø)` | | | [.../spanner/migrations/schema/SessionBasedMapper.java](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1633?src=pr&el=tree&filepath=v2%2Fspanner-common%2Fsrc%2Fmain%2Fjava%2Fcom%2Fgoogle%2Fcloud%2Fteleport%2Fv2%2Fspanner%2Fmigrations%2Fschema%2FSessionBasedMapper.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform#diff-djIvc3Bhbm5lci1jb21tb24vc3JjL21haW4vamF2YS9jb20vZ29vZ2xlL2Nsb3VkL3RlbGVwb3J0L3YyL3NwYW5uZXIvbWlncmF0aW9ucy9zY2hlbWEvU2Vzc2lvbkJhc2VkTWFwcGVyLmphdmE=) | `100.00% <100.00%> (+1.72%)` | :arrow_up: | | [...er/migrations/avro/GenericRecordTypeConvertor.java](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1633?src=pr&el=tree&filepath=v2%2Fspanner-common%2Fsrc%2Fmain%2Fjava%2Fcom%2Fgoogle%2Fcloud%2Fteleport%2Fv2%2Fspanner%2Fmigrations%2Favro%2FGenericRecordTypeConvertor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform#diff-djIvc3Bhbm5lci1jb21tb24vc3JjL21haW4vamF2YS9jb20vZ29vZ2xlL2Nsb3VkL3RlbGVwb3J0L3YyL3NwYW5uZXIvbWlncmF0aW9ucy9hdnJvL0dlbmVyaWNSZWNvcmRUeXBlQ29udmVydG9yLmphdmE=) | `93.00% <90.00%> (-2.42%)` | :arrow_down: | ... and [469 files with indirect coverage changes](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1633/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform)