GoogleCloudPlatform / DataflowTemplates

Cloud Dataflow Google-provided templates for solving in-Cloud data tasks
https://cloud.google.com/dataflow/docs/guides/templates/provided-templates
Apache License 2.0
1.16k stars 978 forks source link

add KMS key support to BigQueryConverters #2024

Open Polber opened 6 days ago

Polber commented 6 days ago

This PR adds support for encrypting temporary table created during BigQueryIO read for any templates that use the BigQueryConverters class (v2) to read from BigQuery.

This only applies when using query to read from the BigQuery source

codecov[bot] commented 6 days ago

Codecov Report

Attention: Patch coverage is 0% with 3 lines in your changes missing coverage. Please review.

Project coverage is 45.41%. Comparing base (ff49726) to head (48dbf87). Report is 2 commits behind head on main.

Files with missing lines Patch % Lines
...oud/teleport/v2/transforms/BigQueryConverters.java 0.00% 3 Missing :warning:
Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #2024 +/- ## ============================================ - Coverage 45.41% 45.41% -0.01% - Complexity 3677 4001 +324 ============================================ Files 842 842 Lines 49970 49970 Branches 5261 5262 +1 ============================================ - Hits 22696 22695 -1 Misses 25605 25605 - Partials 1669 1670 +1 ``` | [Components](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/2024/components?src=pr&el=components&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform) | Coverage Δ | | |---|---|---| | [spanner-templates](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/2024/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform) | `66.71% <ø> (-0.01%)` | :arrow_down: | | [spanner-import-export](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/2024/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform) | `64.18% <ø> (-0.03%)` | :arrow_down: | | [spanner-live-forward-migration](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/2024/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform) | `75.88% <ø> (ø)` | | | [spanner-live-reverse-replication](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/2024/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform) | `76.65% <ø> (ø)` | | | [spanner-bulk-migration](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/2024/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform) | `86.37% <ø> (ø)` | | | [Files with missing lines](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/2024?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform) | Coverage Δ | | |---|---|---| | [...oud/teleport/v2/transforms/BigQueryConverters.java](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/2024?src=pr&el=tree&filepath=v2%2Fcommon%2Fsrc%2Fmain%2Fjava%2Fcom%2Fgoogle%2Fcloud%2Fteleport%2Fv2%2Ftransforms%2FBigQueryConverters.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform#diff-djIvY29tbW9uL3NyYy9tYWluL2phdmEvY29tL2dvb2dsZS9jbG91ZC90ZWxlcG9ydC92Mi90cmFuc2Zvcm1zL0JpZ1F1ZXJ5Q29udmVydGVycy5qYXZh) | `50.21% <0.00%> (ø)` | | ... and [1 file with indirect coverage changes](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/2024/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform)

🚨 Try these New Features: