$ is default delimiter for custom dq query but $ is being escaped by databricks runtime while insertion.
Fix: Changing the default delimiter to @
The regex pattern for capturing the Agg DQ results is failing when the expression has spaces in between. Eg: sum(sales) > 100
Fix: Changing the regex to handle spaces
The summary row dq results catures both the failed and passed row dq rules. But the detailed stats enhancement was programmed with the assumtion that only failed results are captured.
Fix: Handling the pass status scenario in the get_row_dq_detailed_stats method
Motivation and Context
How Has This Been Tested?
Tested with the sample data
Screenshots (if appropriate):
Types of changes
[x] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to change)
Checklist:
[x] My code follows the code style of this project.
[ ] My change requires a change to the documentation.
Description
Related Issue
Bugs
$ is default delimiter for custom dq query but $ is being escaped by databricks runtime while insertion. Fix: Changing the default delimiter to @
The regex pattern for capturing the Agg DQ results is failing when the expression has spaces in between. Eg: sum(sales) > 100 Fix: Changing the regex to handle spaces
The summary row dq results catures both the failed and passed row dq rules. But the detailed stats enhancement was programmed with the assumtion that only failed results are captured. Fix: Handling the pass status scenario in the get_row_dq_detailed_stats method
Motivation and Context
How Has This Been Tested?
Tested with the sample data
Screenshots (if appropriate):
Types of changes
Checklist: