search

GoogleCloudPlatform / DataflowTemplates

Cloud Dataflow Google-provided templates for solving in-Cloud data tasks
https://cloud.google.com/dataflow/docs/guides/templates/provided-templates
Apache License 2.0
1.11k stars 931 forks source link

Add Kafka read topic and kafka write topic pickers replacing kafka topic picker #1676

Closed AnandInguva closed 2 weeks ago

codecov[bot] commented 3 weeks ago

Codecov Report

Attention: Patch coverage is 85.71429% with 4 lines in your changes missing coverage. Please review.

Project coverage is 41.17%. Comparing base (7e3e73a) to head (27b7609). Report is 17 commits behind head on main.

:exclamation: Current head 27b7609 differs from pull request most recent head b23f13f

Please upload reports for the commit b23f13f to get more accurate results.

Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #1676 +/- ## ========================================= Coverage 41.16% 41.17% + Complexity 2928 2926 -2 ========================================= Files 764 764 Lines 44448 44472 +24 Branches 4760 4761 +1 ========================================= + Hits 18299 18311 +12 - Misses 24607 24617 +10 - Partials 1542 1544 +2 ``` | [Components](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1676/components?src=pr&el=components&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform) | Coverage Δ | | |---|---|---| | [spanner-templates](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1676/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform) | `61.37% <ø> (-0.03%)` | :arrow_down: | | [spanner-import-export](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1676/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform) | `64.36% <ø> (-0.07%)` | :arrow_down: | | [spanner-live-forward-migration](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1676/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform) | `73.87% <ø> (ø)` | | | [spanner-live-reverse-replication](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1676/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform) | `49.70% <ø> (ø)` | | | [spanner-bulk-migration](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1676/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform) | `82.06% <ø> (ø)` | | | [Files](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1676?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform) | Coverage Δ | | |---|---|---| | [...gle/cloud/teleport/metadata/TemplateParameter.java](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1676?src=pr&el=tree&filepath=metadata%2Fsrc%2Fmain%2Fjava%2Fcom%2Fgoogle%2Fcloud%2Fteleport%2Fmetadata%2FTemplateParameter.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform#diff-bWV0YWRhdGEvc3JjL21haW4vamF2YS9jb20vZ29vZ2xlL2Nsb3VkL3RlbGVwb3J0L21ldGFkYXRhL1RlbXBsYXRlUGFyYW1ldGVyLmphdmE=) | `0.00% <ø> (ø)` | | | [...le/cloud/teleport/metadata/util/MetadataUtils.java](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1676?src=pr&el=tree&filepath=metadata%2Fsrc%2Fmain%2Fjava%2Fcom%2Fgoogle%2Fcloud%2Fteleport%2Fmetadata%2Futil%2FMetadataUtils.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform#diff-bWV0YWRhdGEvc3JjL21haW4vamF2YS9jb20vZ29vZ2xlL2Nsb3VkL3RlbGVwb3J0L21ldGFkYXRhL3V0aWwvTWV0YWRhdGFVdGlscy5qYXZh) | `50.00% <ø> (ø)` | | | [.../teleport/plugin/model/ImageSpecParameterType.java](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1676?src=pr&el=tree&filepath=plugins%2Fcore-plugin%2Fsrc%2Fmain%2Fjava%2Fcom%2Fgoogle%2Fcloud%2Fteleport%2Fplugin%2Fmodel%2FImageSpecParameterType.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform#diff-cGx1Z2lucy9jb3JlLXBsdWdpbi9zcmMvbWFpbi9qYXZhL2NvbS9nb29nbGUvY2xvdWQvdGVsZXBvcnQvcGx1Z2luL21vZGVsL0ltYWdlU3BlY1BhcmFtZXRlclR5cGUuamF2YQ==) | `100.00% <100.00%> (ø)` | | | [...ud/teleport/v2/kafka/options/KafkaReadOptions.java](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1676?src=pr&el=tree&filepath=v2%2Fkafka-common%2Fsrc%2Fmain%2Fjava%2Fcom%2Fgoogle%2Fcloud%2Fteleport%2Fv2%2Fkafka%2Foptions%2FKafkaReadOptions.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform#diff-djIva2Fma2EtY29tbW9uL3NyYy9tYWluL2phdmEvY29tL2dvb2dsZS9jbG91ZC90ZWxlcG9ydC92Mi9rYWZrYS9vcHRpb25zL0thZmthUmVhZE9wdGlvbnMuamF2YQ==) | `0.00% <ø> (ø)` | | | [...le/cloud/teleport/v2/templates/KafkaToGcsFlex.java](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1676?src=pr&el=tree&filepath=v2%2Fkafka-to-gcs%2Fsrc%2Fmain%2Fjava%2Fcom%2Fgoogle%2Fcloud%2Fteleport%2Fv2%2Ftemplates%2FKafkaToGcsFlex.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform#diff-djIva2Fma2EtdG8tZ2NzL3NyYy9tYWluL2phdmEvY29tL2dvb2dsZS9jbG91ZC90ZWxlcG9ydC92Mi90ZW1wbGF0ZXMvS2Fma2FUb0djc0ZsZXguamF2YQ==) | `0.00% <ø> (ø)` | | | [...loud/teleport/plugin/model/ImageSpecParameter.java](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1676?src=pr&el=tree&filepath=plugins%2Fcore-plugin%2Fsrc%2Fmain%2Fjava%2Fcom%2Fgoogle%2Fcloud%2Fteleport%2Fplugin%2Fmodel%2FImageSpecParameter.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform#diff-cGx1Z2lucy9jb3JlLXBsdWdpbi9zcmMvbWFpbi9qYXZhL2NvbS9nb29nbGUvY2xvdWQvdGVsZXBvcnQvcGx1Z2luL21vZGVsL0ltYWdlU3BlY1BhcmFtZXRlci5qYXZh) | `43.91% <84.61%> (+1.31%)` | :arrow_up: | ... and [4 files with indirect coverage changes](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1676/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform)
AnandInguva commented 3 weeks ago

We should not be replacing KafkaTopic, since we are not dropping support for the older parameter, just adding support for the new ones. So I think KafkaTopic, KafkaReadTopic and KafkaWriteTopic should all be present.

KafkaReadTopic is the new KafkaTopic now. since we don't use KafkaTopic anywhere, It would be better to remove it. From the UX, this is not a breaking change.