search

GoogleCloudPlatform / DataflowTemplates

Cloud Dataflow Google-provided templates for solving in-Cloud data tasks
https://cloud.google.com/dataflow/docs/guides/templates/provided-templates
Apache License 2.0
1.14k stars 950 forks source link

[Do not merge] Beam 2.58.0rc2 validation #1761

Closed Abacn closed 1 month ago

codecov[bot] commented 1 month ago

Codecov Report

Attention: Patch coverage is 0% with 17 lines in your changes missing coverage. Please review.

Project coverage is 42.66%. Comparing base (483fbfd) to head (5f1c59c).

Files Patch % Lines
...cloud/teleport/v2/templates/SpannerToBigQuery.java 0.00% 7 Missing :warning:
...t/v2/transforms/PythonExternalTextTransformer.java 0.00% 2 Missing :warning:
...asticsearch/templates/BigQueryToElasticsearch.java 0.00% 1 Missing :warning:
...v2/elasticsearch/templates/GCSToElasticsearch.java 0.00% 1 Missing :warning:
...elasticsearch/templates/PubSubToElasticsearch.java 0.00% 1 Missing :warning:
.../cloud/teleport/v2/templates/PubSubToBigQuery.java 0.00% 1 Missing :warning:
.../cloud/teleport/v2/templates/TextIOToBigQuery.java 0.00% 1 Missing :warning:
...teleport/v2/templates/TextToBigQueryStreaming.java 0.00% 1 Missing :warning:
...d/teleport/v2/templates/PubsubProtoToBigQuery.java 0.00% 1 Missing :warning:
...e/cloud/teleport/v2/templates/PubSubToMongoDB.java 0.00% 1 Missing :warning:
Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #1761 +/- ## ============================================ - Coverage 42.66% 42.66% -0.01% - Complexity 3379 3382 +3 ============================================ Files 815 815 Lines 47693 47705 +12 Branches 5109 5111 +2 ============================================ + Hits 20347 20352 +5 - Misses 25688 25696 +8 + Partials 1658 1657 -1 ``` | [Components](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1761/components?src=pr&el=components&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform) | Coverage Δ | | |---|---|---| | [spanner-templates](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1761/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform) | `64.57% <ø> (+0.01%)` | :arrow_up: | | [spanner-import-export](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1761/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform) | `64.42% <ø> (+0.04%)` | :arrow_up: | | [spanner-live-forward-migration](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1761/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform) | `75.00% <ø> (ø)` | | | [spanner-live-reverse-replication](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1761/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform) | `51.96% <ø> (ø)` | | | [spanner-bulk-migration](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1761/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform) | `83.46% <ø> (ø)` | | | [Files](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1761?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform) | Coverage Δ | | |---|---|---| | [...asticsearch/templates/BigQueryToElasticsearch.java](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1761?src=pr&el=tree&filepath=v2%2Fgooglecloud-to-elasticsearch%2Fsrc%2Fmain%2Fjava%2Fcom%2Fgoogle%2Fcloud%2Fteleport%2Fv2%2Felasticsearch%2Ftemplates%2FBigQueryToElasticsearch.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform#diff-djIvZ29vZ2xlY2xvdWQtdG8tZWxhc3RpY3NlYXJjaC9zcmMvbWFpbi9qYXZhL2NvbS9nb29nbGUvY2xvdWQvdGVsZXBvcnQvdjIvZWxhc3RpY3NlYXJjaC90ZW1wbGF0ZXMvQmlnUXVlcnlUb0VsYXN0aWNzZWFyY2guamF2YQ==) | `0.00% <0.00%> (ø)` | | | [...v2/elasticsearch/templates/GCSToElasticsearch.java](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1761?src=pr&el=tree&filepath=v2%2Fgooglecloud-to-elasticsearch%2Fsrc%2Fmain%2Fjava%2Fcom%2Fgoogle%2Fcloud%2Fteleport%2Fv2%2Felasticsearch%2Ftemplates%2FGCSToElasticsearch.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform#diff-djIvZ29vZ2xlY2xvdWQtdG8tZWxhc3RpY3NlYXJjaC9zcmMvbWFpbi9qYXZhL2NvbS9nb29nbGUvY2xvdWQvdGVsZXBvcnQvdjIvZWxhc3RpY3NlYXJjaC90ZW1wbGF0ZXMvR0NTVG9FbGFzdGljc2VhcmNoLmphdmE=) | `12.65% <0.00%> (-0.17%)` | :arrow_down: | | [...elasticsearch/templates/PubSubToElasticsearch.java](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1761?src=pr&el=tree&filepath=v2%2Fgooglecloud-to-elasticsearch%2Fsrc%2Fmain%2Fjava%2Fcom%2Fgoogle%2Fcloud%2Fteleport%2Fv2%2Felasticsearch%2Ftemplates%2FPubSubToElasticsearch.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform#diff-djIvZ29vZ2xlY2xvdWQtdG8tZWxhc3RpY3NlYXJjaC9zcmMvbWFpbi9qYXZhL2NvbS9nb29nbGUvY2xvdWQvdGVsZXBvcnQvdjIvZWxhc3RpY3NlYXJjaC90ZW1wbGF0ZXMvUHViU3ViVG9FbGFzdGljc2VhcmNoLmphdmE=) | `10.93% <0.00%> (-0.18%)` | :arrow_down: | | [.../cloud/teleport/v2/templates/PubSubToBigQuery.java](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1761?src=pr&el=tree&filepath=v2%2Fgooglecloud-to-googlecloud%2Fsrc%2Fmain%2Fjava%2Fcom%2Fgoogle%2Fcloud%2Fteleport%2Fv2%2Ftemplates%2FPubSubToBigQuery.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform#diff-djIvZ29vZ2xlY2xvdWQtdG8tZ29vZ2xlY2xvdWQvc3JjL21haW4vamF2YS9jb20vZ29vZ2xlL2Nsb3VkL3RlbGVwb3J0L3YyL3RlbXBsYXRlcy9QdWJTdWJUb0JpZ1F1ZXJ5LmphdmE=) | `34.40% <0.00%> (-0.28%)` | :arrow_down: | | [.../cloud/teleport/v2/templates/TextIOToBigQuery.java](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1761?src=pr&el=tree&filepath=v2%2Fgooglecloud-to-googlecloud%2Fsrc%2Fmain%2Fjava%2Fcom%2Fgoogle%2Fcloud%2Fteleport%2Fv2%2Ftemplates%2FTextIOToBigQuery.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform#diff-djIvZ29vZ2xlY2xvdWQtdG8tZ29vZ2xlY2xvdWQvc3JjL21haW4vamF2YS9jb20vZ29vZ2xlL2Nsb3VkL3RlbGVwb3J0L3YyL3RlbXBsYXRlcy9UZXh0SU9Ub0JpZ1F1ZXJ5LmphdmE=) | `67.10% <0.00%> (-0.90%)` | :arrow_down: | | [...teleport/v2/templates/TextToBigQueryStreaming.java](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1761?src=pr&el=tree&filepath=v2%2Fgooglecloud-to-googlecloud%2Fsrc%2Fmain%2Fjava%2Fcom%2Fgoogle%2Fcloud%2Fteleport%2Fv2%2Ftemplates%2FTextToBigQueryStreaming.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform#diff-djIvZ29vZ2xlY2xvdWQtdG8tZ29vZ2xlY2xvdWQvc3JjL21haW4vamF2YS9jb20vZ29vZ2xlL2Nsb3VkL3RlbGVwb3J0L3YyL3RlbXBsYXRlcy9UZXh0VG9CaWdRdWVyeVN0cmVhbWluZy5qYXZh) | `15.31% <0.00%> (-0.14%)` | :arrow_down: | | [...d/teleport/v2/templates/PubsubProtoToBigQuery.java](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1761?src=pr&el=tree&filepath=v2%2Fpubsub-binary-to-bigquery%2Fsrc%2Fmain%2Fjava%2Fcom%2Fgoogle%2Fcloud%2Fteleport%2Fv2%2Ftemplates%2FPubsubProtoToBigQuery.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform#diff-djIvcHVic3ViLWJpbmFyeS10by1iaWdxdWVyeS9zcmMvbWFpbi9qYXZhL2NvbS9nb29nbGUvY2xvdWQvdGVsZXBvcnQvdjIvdGVtcGxhdGVzL1B1YnN1YlByb3RvVG9CaWdRdWVyeS5qYXZh) | `12.39% <0.00%> (-0.11%)` | :arrow_down: | | [...e/cloud/teleport/v2/templates/PubSubToMongoDB.java](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1761?src=pr&el=tree&filepath=v2%2Fpubsub-to-mongodb%2Fsrc%2Fmain%2Fjava%2Fcom%2Fgoogle%2Fcloud%2Fteleport%2Fv2%2Ftemplates%2FPubSubToMongoDB.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform#diff-djIvcHVic3ViLXRvLW1vbmdvZGIvc3JjL21haW4vamF2YS9jb20vZ29vZ2xlL2Nsb3VkL3RlbGVwb3J0L3YyL3RlbXBsYXRlcy9QdWJTdWJUb01vbmdvREIuamF2YQ==) | `34.58% <0.00%> (-0.27%)` | :arrow_down: | | [...t/v2/transforms/PythonExternalTextTransformer.java](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1761?src=pr&el=tree&filepath=v2%2Fcommon%2Fsrc%2Fmain%2Fjava%2Fcom%2Fgoogle%2Fcloud%2Fteleport%2Fv2%2Ftransforms%2FPythonExternalTextTransformer.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform#diff-djIvY29tbW9uL3NyYy9tYWluL2phdmEvY29tL2dvb2dsZS9jbG91ZC90ZWxlcG9ydC92Mi90cmFuc2Zvcm1zL1B5dGhvbkV4dGVybmFsVGV4dFRyYW5zZm9ybWVyLmphdmE=) | `0.00% <0.00%> (ø)` | | | [...cloud/teleport/v2/templates/SpannerToBigQuery.java](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1761?src=pr&el=tree&filepath=v2%2Fgooglecloud-to-googlecloud%2Fsrc%2Fmain%2Fjava%2Fcom%2Fgoogle%2Fcloud%2Fteleport%2Fv2%2Ftemplates%2FSpannerToBigQuery.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform#diff-djIvZ29vZ2xlY2xvdWQtdG8tZ29vZ2xlY2xvdWQvc3JjL21haW4vamF2YS9jb20vZ29vZ2xlL2Nsb3VkL3RlbGVwb3J0L3YyL3RlbXBsYXRlcy9TcGFubmVyVG9CaWdRdWVyeS5qYXZh) | `0.00% <0.00%> (ø)` | | ... and [3 files with indirect coverage changes](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1761/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform)
Abacn commented 1 month ago

still not be able to validate Python UDF tests. After override customBeamRequirement, template launching passed, but worker stuck at

Checking docker image integrity of gcr.io/cloud-dataflow/v1beta3/beam_python3.11_sdk:2.58.0rc2 at "container_images.go:106"

"failed to \"StartContainer\" for \"sdk-0-0\" with ImagePullBackOff: \"Back-off pulling image \\\"gcr.io/cloud-dataflow/v1beta3/beam_python3.11_sdk:2.58.0rc2\\\" at "pod_workers.go:951"

Otherwise passed, including Yaml