GoogleCloudPlatform / DataflowTemplates

Cloud Dataflow Google-provided templates for solving in-Cloud data tasks
https://cloud.google.com/dataflow/docs/guides/templates/provided-templates
Apache License 2.0
1.12k stars 936 forks source link

Added support for advanced transformation in reverse replication #1655

Closed shreyakhajanchi closed 1 month ago

codecov[bot] commented 1 month ago

Codecov Report

Attention: Patch coverage is 24.35897% with 59 lines in your changes missing coverage. Please review.

Project coverage is 45.44%. Comparing base (3a6d1bb) to head (8d83429). Report is 73 commits behind head on main.

:exclamation: Current head 8d83429 differs from pull request most recent head c44d3d2

Please upload reports for the commit c44d3d2 to get more accurate results.

Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #1655 +/- ## ============================================ + Coverage 41.17% 45.44% +4.26% + Complexity 3196 717 -2479 ============================================ Files 764 301 -463 Lines 44449 16245 -28204 Branches 4759 1618 -3141 ============================================ - Hits 18304 7382 -10922 + Misses 24604 8321 -16283 + Partials 1541 542 -999 ``` | [Components](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1655/components?src=pr&el=components&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform) | Coverage Δ | | |---|---|---| | [spanner-templates](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1655/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform) | `58.67% <24.35%> (-2.74%)` | :arrow_down: | | [spanner-import-export](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1655/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform) | `∅ <ø> (∅)` | | | [spanner-live-forward-migration](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1655/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform) | `74.04% <100.00%> (+0.15%)` | :arrow_up: | | [spanner-live-reverse-replication](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1655/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform) | `49.25% <24.35%> (-0.47%)` | :arrow_down: | | [spanner-bulk-migration](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1655/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform) | `82.17% <100.00%> (+0.10%)` | :arrow_up: | | [Files](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1655?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform) | Coverage Δ | | |---|---|---| | [...grations/convertors/ChangeEventToMapConvertor.java](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1655?src=pr&el=tree&filepath=v2%2Fspanner-common%2Fsrc%2Fmain%2Fjava%2Fcom%2Fgoogle%2Fcloud%2Fteleport%2Fv2%2Fspanner%2Fmigrations%2Fconvertors%2FChangeEventToMapConvertor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform#diff-djIvc3Bhbm5lci1jb21tb24vc3JjL21haW4vamF2YS9jb20vZ29vZ2xlL2Nsb3VkL3RlbGVwb3J0L3YyL3NwYW5uZXIvbWlncmF0aW9ucy9jb252ZXJ0b3JzL0NoYW5nZUV2ZW50VG9NYXBDb252ZXJ0b3IuamF2YQ==) | `90.76% <100.00%> (+3.81%)` | :arrow_up: | | [...gle/cloud/teleport/v2/templates/GCSToSourceDb.java](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1655?src=pr&el=tree&filepath=v2%2Fgcs-to-sourcedb%2Fsrc%2Fmain%2Fjava%2Fcom%2Fgoogle%2Fcloud%2Fteleport%2Fv2%2Ftemplates%2FGCSToSourceDb.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform#diff-djIvZ2NzLXRvLXNvdXJjZWRiL3NyYy9tYWluL2phdmEvY29tL2dvb2dsZS9jbG91ZC90ZWxlcG9ydC92Mi90ZW1wbGF0ZXMvR0NTVG9Tb3VyY2VEYi5qYXZh) | `0.00% <0.00%> (ø)` | | | [...t/v2/templates/transforms/GcsToSourceStreamer.java](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1655?src=pr&el=tree&filepath=v2%2Fgcs-to-sourcedb%2Fsrc%2Fmain%2Fjava%2Fcom%2Fgoogle%2Fcloud%2Fteleport%2Fv2%2Ftemplates%2Ftransforms%2FGcsToSourceStreamer.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform#diff-djIvZ2NzLXRvLXNvdXJjZWRiL3NyYy9tYWluL2phdmEvY29tL2dvb2dsZS9jbG91ZC90ZWxlcG9ydC92Mi90ZW1wbGF0ZXMvdHJhbnNmb3Jtcy9HY3NUb1NvdXJjZVN0cmVhbWVyLmphdmE=) | `0.00% <0.00%> (ø)` | | | [...lates/processing/handler/InputRecordProcessor.java](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1655?src=pr&el=tree&filepath=v2%2Fgcs-to-sourcedb%2Fsrc%2Fmain%2Fjava%2Fcom%2Fgoogle%2Fcloud%2Fteleport%2Fv2%2Ftemplates%2Fprocessing%2Fhandler%2FInputRecordProcessor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform#diff-djIvZ2NzLXRvLXNvdXJjZWRiL3NyYy9tYWluL2phdmEvY29tL2dvb2dsZS9jbG91ZC90ZWxlcG9ydC92Mi90ZW1wbGF0ZXMvcHJvY2Vzc2luZy9oYW5kbGVyL0lucHV0UmVjb3JkUHJvY2Vzc29yLmphdmE=) | `0.00% <0.00%> (ø)` | | | [...rocessing/handler/GCSToSourceStreamingHandler.java](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1655?src=pr&el=tree&filepath=v2%2Fgcs-to-sourcedb%2Fsrc%2Fmain%2Fjava%2Fcom%2Fgoogle%2Fcloud%2Fteleport%2Fv2%2Ftemplates%2Fprocessing%2Fhandler%2FGCSToSourceStreamingHandler.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform#diff-djIvZ2NzLXRvLXNvdXJjZWRiL3NyYy9tYWluL2phdmEvY29tL2dvb2dsZS9jbG91ZC90ZWxlcG9ydC92Mi90ZW1wbGF0ZXMvcHJvY2Vzc2luZy9oYW5kbGVyL0dDU1RvU291cmNlU3RyZWFtaW5nSGFuZGxlci5qYXZh) | `0.00% <0.00%> (ø)` | | ... and [482 files with indirect coverage changes](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1655/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform)
aksharauke commented 1 month ago

To me the UT coverage looks okay. The un-covered lines are mainly setup methods and not business logic. It should be okay, in my view, to test these via IT. Hence should be okay to add override flags for migration.