GoogleCloudPlatform / dlp-dataflow-deidentification

Multi Cloud Data Tokenization Solution By Using Dataflow and Cloud DLP
Apache License 2.0
89 stars 53 forks source link

Changes to make queryPath optional #162

Open dup05 opened 1 year ago

dup05 commented 1 year ago

Summary (Short summary of what is being done) :

Simplify REID workflow by making the query parameter optional

Description (Describe in detail the fix made) :

The REID workflow required users to compulsarily provide SQL query to read from the BigQuery. This involved manual steps to create a GCS object containing sql query and passing its path in the pipeline parameter. With this change, the queryPath parameter has been made optional and users can run REID pipeline without needing to provide a SQL query.

Bug ID (if any) :

b/293427326

Public Documentation (if any) :

TESTED (Test Cases with scenario and description - must have 1 positive and 1 negative scenario) :

1. Tested REID without passing queryPath and verified the results 2. Tested REID by passing the query
codecov[bot] commented 1 year ago

Codecov Report

Merging #162 (01f8279) into master (0139d92) will decrease coverage by 0.03%. Report is 12 commits behind head on master. The diff coverage is 0.00%.

@@             Coverage Diff              @@
##             master     #162      +/-   ##
============================================
- Coverage     12.45%   12.43%   -0.03%     
  Complexity       63       63              
============================================
  Files            53       53              
  Lines          2480     2484       +4     
  Branches        207      208       +1     
============================================
  Hits            309      309              
- Misses         2152     2156       +4     
  Partials         19       19              
Files Coverage Δ
...n/DLPTextToBigQueryStreamingV2PipelineOptions.java 0.00% <ø> (ø)
...ava/com/google/swarm/tokenization/common/Util.java 30.24% <0.00%> (-0.33%) :arrow_down:
...arm/tokenization/common/BigQueryReadTransform.java 0.00% <0.00%> (ø)

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more