GoogleCloudPlatform / DataflowTemplates

Cloud Dataflow Google-provided templates for solving in-Cloud data tasks
https://cloud.google.com/dataflow/docs/guides/templates/provided-templates
Apache License 2.0
1.12k stars 936 forks source link

base64 handling simplified #1665

Closed aksharauke closed 1 month ago

aksharauke commented 1 month ago

For BYTES when Spanner stores it as base64 encoded, no need to decode it and escape - rather use the built in constructs to handle base64. Similar use of built in constructs to handle hex.

codecov[bot] commented 1 month ago

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 45.54%. Comparing base (e678faf) to head (e8472ad). Report is 23 commits behind head on main.

Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #1665 +/- ## ============================================ + Coverage 41.25% 45.54% +4.28% + Complexity 2928 717 -2211 ============================================ Files 763 301 -462 Lines 44372 16164 -28208 Branches 4752 1606 -3146 ============================================ - Hits 18307 7362 -10945 + Misses 24521 8262 -16259 + Partials 1544 540 -1004 ``` | [Components](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1665/components?src=pr&el=components&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform) | Coverage Δ | | |---|---|---| | [spanner-templates](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1665/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform) | `59.10% <100.00%> (-2.27%)` | :arrow_down: | | [spanner-import-export](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1665/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform) | `∅ <ø> (∅)` | | | [spanner-live-forward-migration](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1665/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform) | `73.88% <ø> (ø)` | | | [spanner-live-reverse-replication](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1665/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform) | `49.72% <100.00%> (+0.05%)` | :arrow_up: | | [spanner-bulk-migration](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1665/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform) | `82.07% <ø> (ø)` | | | [Files](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1665?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform) | Coverage Δ | | |---|---|---| | [...port/v2/templates/processing/dml/DMLGenerator.java](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1665?src=pr&el=tree&filepath=v2%2Fgcs-to-sourcedb%2Fsrc%2Fmain%2Fjava%2Fcom%2Fgoogle%2Fcloud%2Fteleport%2Fv2%2Ftemplates%2Fprocessing%2Fdml%2FDMLGenerator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform#diff-djIvZ2NzLXRvLXNvdXJjZWRiL3NyYy9tYWluL2phdmEvY29tL2dvb2dsZS9jbG91ZC90ZWxlcG9ydC92Mi90ZW1wbGF0ZXMvcHJvY2Vzc2luZy9kbWwvRE1MR2VuZXJhdG9yLmphdmE=) | `78.68% <100.00%> (+2.37%)` | :arrow_up: | ... and [478 files with indirect coverage changes](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1665/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform)