Nike-Inc / spark-expectations

A Python Library to support running data quality rules while the spark job is running⚡
https://engineering.nike.com/spark-expectations
Apache License 2.0
161 stars 37 forks source link

Creating a detailed stats table for capturing details on the execution of dq rules. #80

Closed vigneshwarrvenkat closed 6 months ago

vigneshwarrvenkat commented 6 months ago

detailed stats table for capturing details on the execution of dq rules

Description

https://github.com/Nike-Inc/spark-expectations/issues/55

Related Issue

https://github.com/Nike-Inc/spark-expectations/issues/55

Motivation and Context

As of now, the stats table details are not sufficient to get detailed insights on the rules executed. This PR provides more details using a detailed stats table on the execution of all the rules in a relational format.

How Has This Been Tested?

The testing was done in local env with the sample data provided with this framework.

Screenshots (if appropriate):

image image image

Types of changes

Checklist:

codecov[bot] commented 6 months ago

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 100.00%. Comparing base (a725aaa) to head (58189c7).

Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #80 +/- ## ========================================== Coverage 100.00% 100.00% ========================================== Files 22 22 Lines 1490 1809 +319 ========================================== + Hits 1490 1809 +319 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.