Replace Spark SQL isNull check with Spark Scala based DSL - Githubissues

awslabs / deequ

Deequ is a library built on top of Apache Spark for defining "unit tests for data", which measure data quality in large datasets.

Apache License 2.0

3.27k stars 536 forks source link

Replace Spark SQL isNull check with Spark Scala based DSL #493

Closed rdsharma26 closed 1 year ago

rdsharma26 commented 1 year ago

Issue

The column profiling for min/max lengths for a string column is not working when the column name has a space in it.

Description of changes:

Replaced Spark SQL isNull check with Spark Scala based DSL
This is to ensure columns with spaces in their names get their names escaped correctly in the where condition.
Added a test to verify.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.