Nike-Inc / spark-expectations

A Python Library to support running data quality rules while the spark job is running⚡
https://engineering.nike.com/spark-expectations
Apache License 2.0
148 stars 32 forks source link

Updating the flow diagram and enabling summarized row dq to capture all results (fail/pass) as part of PR - 80 #90

Closed vigneshwarrvenkat closed 1 month ago

vigneshwarrvenkat commented 1 month ago

Description

  1. Updating the flow diagram as part of PR - 80 : https://github.com/Nike-Inc/spark-expectations/pull/80
  2. Enabling summarized row dq to capture all results (fail/pass)

Related Issue

summarized row dq was capturing the results only when there is a failure in Row DQ. This results in details getting skipped in the detailed stats table when there is no failure in Row DQ.

Motivation and Context

Enabling this feature would provide full detailed output in the detailed stats table even if there is no failure in row dq.

How Has This Been Tested?

Tested with the current sample data.

Screenshots (if appropriate):

Types of changes

Checklist: