Verify that non key columns exist in each dataset

awslabs / deequ

Deequ is a library built on top of Apache Spark for defining "unit tests for data", which measure data quality in large datasets.

Apache License 2.0

3.32k stars 539 forks source link

Closed rdsharma26 closed 1 year ago

rdsharma26 commented 1 year ago

Description of changes:

The non existence of non key columns was resulting in a Spark SQL exception, instead of a graceful "ComparisonFailed" return value.
In a future PR, will consolidate all the column validation logic into one single place.

Issue #, if available: N/A

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.