search

GoogleCloudPlatform / DataflowTemplates

Cloud Dataflow Google-provided templates for solving in-Cloud data tasks
https://cloud.google.com/dataflow/docs/guides/templates/provided-templates
Apache License 2.0
1.16k stars 978 forks source link

feat: treat empty strings as null values from csv imports #1996

Closed Emrehzl94 closed 2 weeks ago

Emrehzl94 commented 2 weeks ago

This PR enables empty strings in CSV files to be treated as null values during import and adds an integration test to verify this behavior.

codecov[bot] commented 2 weeks ago

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 45.31%. Comparing base (1bef5ff) to head (892cc92). Report is 1 commits behind head on main.

Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #1996 +/- ## ============================================ + Coverage 45.30% 45.31% +0.01% - Complexity 3631 3637 +6 ============================================ Files 836 836 Lines 49922 49922 Branches 5260 5260 ============================================ + Hits 22615 22624 +9 + Misses 25631 25620 -11 - Partials 1676 1678 +2 ``` | [Components](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1996/components?src=pr&el=components&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform) | Coverage Δ | | |---|---|---| | [spanner-templates](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1996/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform) | `66.64% <ø> (-0.02%)` | :arrow_down: | | [spanner-import-export](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1996/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform) | `64.12% <ø> (-0.07%)` | :arrow_down: | | [spanner-live-forward-migration](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1996/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform) | `75.88% <ø> (ø)` | | | [spanner-live-reverse-replication](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1996/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform) | `76.65% <ø> (ø)` | | | [spanner-bulk-migration](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1996/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform) | `86.35% <ø> (ø)` | | | [Files with missing lines](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1996?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform) | Coverage Δ | | |---|---|---| | [...ud/teleport/v2/neo4j/model/helpers/CsvSources.java](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1996?src=pr&el=tree&filepath=v2%2Fgooglecloud-to-neo4j%2Fsrc%2Fmain%2Fjava%2Fcom%2Fgoogle%2Fcloud%2Fteleport%2Fv2%2Fneo4j%2Fmodel%2Fhelpers%2FCsvSources.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform#diff-djIvZ29vZ2xlY2xvdWQtdG8tbmVvNGovc3JjL21haW4vamF2YS9jb20vZ29vZ2xlL2Nsb3VkL3RlbGVwb3J0L3YyL25lbzRqL21vZGVsL2hlbHBlcnMvQ3N2U291cmNlcy5qYXZh) | `91.66% <100.00%> (+75.00%)` | :arrow_up: | | [...oogle/cloud/teleport/v2/neo4j/utils/BeamUtils.java](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1996?src=pr&el=tree&filepath=v2%2Fgooglecloud-to-neo4j%2Fsrc%2Fmain%2Fjava%2Fcom%2Fgoogle%2Fcloud%2Fteleport%2Fv2%2Fneo4j%2Futils%2FBeamUtils.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform#diff-djIvZ29vZ2xlY2xvdWQtdG8tbmVvNGovc3JjL21haW4vamF2YS9jb20vZ29vZ2xlL2Nsb3VkL3RlbGVwb3J0L3YyL25lbzRqL3V0aWxzL0JlYW1VdGlscy5qYXZh) | `44.30% <100.00%> (ø)` | | ... and [3 files with indirect coverage changes](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1996/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform)
liferoad commented 2 weeks ago

Failed tests are not related to this PR. Merge this now.