search

GoogleCloudPlatform / DataflowTemplates

Cloud Dataflow Google-provided templates for solving in-Cloud data tasks
https://cloud.google.com/dataflow/docs/guides/templates/provided-templates
Apache License 2.0
1.14k stars 950 forks source link

Support FLOAT32 in Spanner Change Streams to BQ #1727

Closed arawind closed 1 month ago

arawind commented 1 month ago

Also fixes the BigQueryDynamicDestinationsTest, adds a new float column test in SpannerChangeStreamsToBigQuery and updates SpannerChangeStreamsToGCS and PubSub integration tests.

Verfied that the changes work by running the BigQueryDynamicDestinationsTest and SpannerChangeStreamsToBigQueryIT tests manually.

Commands for future reference:

To run BigQueryDynamicDestinationsTest:

mvn clean verify -f pom.xml -e -Dcheckstyle.skip -Djib.skip -DskipShade -Dspotless.check.skip -Djacoco.skip -fae -T8 -Dexcluded.spanner.tests="" -DfailIfNoTests=false -pl v2/googlecloud-to-googlecloud,plugins/templates-maven-plugin -am -Dtest=BigQueryDynamicDestinationsTest;

To run SpannerChangeStreamsToBigQueryIT:

mvn clean verify -f pom.xml -e -Dcheckstyle.skip -Djib.skip -DskipShade -Dspotless.check.skip -Djacoco.skip -fae -T8 -Dexcluded.spanner.tests="" -DfailIfNoTests=false -pl v2/googlecloud-to-googlecloud,plugins/templates-maven-plugin -am -Dproject=span-cloud-testing -DspannerInstanceId=arawind-test -DartifactBucket="gs://arawind-spanner-test/" -Dtest=SpannerChangeStreamsToBigQueryIT;
codecov[bot] commented 1 month ago

Codecov Report

Attention: Patch coverage is 0% with 10 lines in your changes missing coverage. Please review.

Project coverage is 42.25%. Comparing base (e19dcbc) to head (c38d63d). Report is 16 commits behind head on main.

Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #1727 +/- ## ============================================ - Coverage 42.27% 42.25% -0.02% + Complexity 3167 3166 -1 ============================================ Files 791 791 Lines 46131 46147 +16 Branches 4934 4938 +4 ============================================ - Hits 19502 19501 -1 - Misses 25039 25052 +13 - Partials 1590 1594 +4 ``` | [Components](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1727/components?src=pr&el=components&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform) | Coverage Δ | | |---|---|---| | [spanner-templates](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1727/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform) | `63.63% <ø> (-0.01%)` | :arrow_down: | | [spanner-import-export](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1727/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform) | `64.42% <ø> (-0.03%)` | :arrow_down: | | [spanner-live-forward-migration](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1727/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform) | `74.97% <ø> (ø)` | | | [spanner-live-reverse-replication](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1727/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform) | `51.44% <ø> (ø)` | | | [spanner-bulk-migration](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1727/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform) | `83.14% <ø> (ø)` | | | [Files](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1727?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform) | Coverage Δ | | |---|---|---| | [...igquery/schemautils/SpannerChangeStreamsUtils.java](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1727?src=pr&el=tree&filepath=v2%2Fgooglecloud-to-googlecloud%2Fsrc%2Fmain%2Fjava%2Fcom%2Fgoogle%2Fcloud%2Fteleport%2Fv2%2Ftemplates%2Fspannerchangestreamstobigquery%2Fschemautils%2FSpannerChangeStreamsUtils.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform#diff-djIvZ29vZ2xlY2xvdWQtdG8tZ29vZ2xlY2xvdWQvc3JjL21haW4vamF2YS9jb20vZ29vZ2xlL2Nsb3VkL3RlbGVwb3J0L3YyL3RlbXBsYXRlcy9zcGFubmVyY2hhbmdlc3RyZWFtc3RvYmlncXVlcnkvc2NoZW1hdXRpbHMvU3Bhbm5lckNoYW5nZVN0cmVhbXNVdGlscy5qYXZh) | `70.89% <0.00%> (-0.54%)` | :arrow_down: | | [...tobigquery/schemautils/SpannerToBigQueryUtils.java](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1727?src=pr&el=tree&filepath=v2%2Fgooglecloud-to-googlecloud%2Fsrc%2Fmain%2Fjava%2Fcom%2Fgoogle%2Fcloud%2Fteleport%2Fv2%2Ftemplates%2Fspannerchangestreamstobigquery%2Fschemautils%2FSpannerToBigQueryUtils.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform#diff-djIvZ29vZ2xlY2xvdWQtdG8tZ29vZ2xlY2xvdWQvc3JjL21haW4vamF2YS9jb20vZ29vZ2xlL2Nsb3VkL3RlbGVwb3J0L3YyL3RlbXBsYXRlcy9zcGFubmVyY2hhbmdlc3RyZWFtc3RvYmlncXVlcnkvc2NoZW1hdXRpbHMvU3Bhbm5lclRvQmlnUXVlcnlVdGlscy5qYXZh) | `73.02% <0.00%> (-4.06%)` | :arrow_down: | ... and [2 files with indirect coverage changes](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1727/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform)
Abacn commented 1 month ago

failed integration test not related. @Polber should we move vspannerchangestreamstobigquery tests to spanner-pr workflow?