GoogleCloudPlatform / DataflowTemplates

Cloud Dataflow Google-provided templates for solving in-Cloud data tasks
https://cloud.google.com/dataflow/docs/guides/templates/provided-templates
Apache License 2.0
1.12k stars 936 forks source link

[BULK] Pass through unsupported types as null #1641

Closed Deep1998 closed 1 month ago

Deep1998 commented 1 month ago

Currently, unsupported types (like spatial types in mysql) throw error and are written to DLQ. This is inconsistent with the live template. This change passes a null values for those types and tries to write the row to Spanner anyways.

codecov[bot] commented 1 month ago

Codecov Report

Attention: Patch coverage is 0% with 3 lines in your changes missing coverage. Please review.

Project coverage is 45.39%. Comparing base (0837b98) to head (6d8e172).

Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #1641 +/- ## ============================================ + Coverage 41.49% 45.39% +3.89% + Complexity 2920 713 -2207 ============================================ Files 754 299 -455 Lines 43977 16128 -27849 Branches 4707 1608 -3099 ============================================ - Hits 18248 7321 -10927 + Misses 24190 8266 -15924 + Partials 1539 541 -998 ``` | [Components](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1641/components?src=pr&el=components&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform) | Coverage Δ | | |---|---|---| | [spanner-templates](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1641/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform) | `58.79% <0.00%> (-2.48%)` | :arrow_down: | | [spanner-import-export](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1641/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform) | `∅ <ø> (∅)` | | | [spanner-live-forward-migration](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1641/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform) | `73.49% <0.00%> (-0.08%)` | :arrow_down: | | [spanner-live-reverse-replication](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1641/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform) | `49.01% <0.00%> (-0.15%)` | :arrow_down: | | [spanner-bulk-migration](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1641/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform) | `81.84% <0.00%> (-0.08%)` | :arrow_down: | | [Files](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1641?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform) | Coverage Δ | | |---|---|---| | [...er/migrations/avro/GenericRecordTypeConvertor.java](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1641?src=pr&el=tree&filepath=v2%2Fspanner-common%2Fsrc%2Fmain%2Fjava%2Fcom%2Fgoogle%2Fcloud%2Fteleport%2Fv2%2Fspanner%2Fmigrations%2Favro%2FGenericRecordTypeConvertor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform#diff-djIvc3Bhbm5lci1jb21tb24vc3JjL21haW4vamF2YS9jb20vZ29vZ2xlL2Nsb3VkL3RlbGVwb3J0L3YyL3NwYW5uZXIvbWlncmF0aW9ucy9hdnJvL0dlbmVyaWNSZWNvcmRUeXBlQ29udmVydG9yLmphdmE=) | `93.28% <0.00%> (-2.14%)` | :arrow_down: | ... and [471 files with indirect coverage changes](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1641/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform)
Deep1998 commented 1 month ago

Changes merged.