performance has been tested on 44 million records. performance has improved write_error_records_final function and action_on_rules function. Additionally, introduced new feature, that has details stats in relational format
Screenshots (if appropriate):
Types of changes
[x] Bug fix (non-breaking change which fixes an issue)
[x] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to change)
Checklist:
[x] My code follows the code style of this project.
[x] My change requires a change to the documentation.
Description
This PR solves two issues: https://github.com/Nike-Inc/spark-expectations/issues/68 and https://github.com/Nike-Inc/spark-expectations/issues/50
Related Issue
Motivation and Context
see issues
How Has This Been Tested?
performance has been tested on 44 million records. performance has improved write_error_records_final function and action_on_rules function. Additionally, introduced new feature, that has details stats in relational format
Screenshots (if appropriate):
Types of changes
Checklist: