[Experimental] Added a function to the Data Synchronization utility for annotating the dataframe with row level results - Githubissues

awslabs / deequ

Deequ is a library built on top of Apache Spark for defining "unit tests for data", which measure data quality in large datasets.

Apache License 2.0

3.32k stars 539 forks source link

[Experimental] Added a function to the Data Synchronization utility for annotating the dataframe with row level results #473

Closed rdsharma26 closed 1 year ago

rdsharma26 commented 1 year ago

Description of changes:

We hash the non key columns from both dataframes and store them as additional columns in each dataframe.
We use the key columns map and the hash columns to join the data frames.
We then check if the key columns and the hash column from the second dataframe are not null. If not null, then the row matches, otherwise it does not.
Added tests to verify the behavior for non-nested and nested data.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.