search

awslabs / python-deequ

Python API for Deequ
Apache License 2.0
691 stars 132 forks source link

checks.py does not support where clause filter of deequ #44

Open rahulgoyal2987 opened 3 years ago

rahulgoyal2987 commented 3 years ago

from pydeequ.checks import from pydeequ.verification import

check = Check(spark, CheckLevel.Warning, "Review Check")

checkResult = VerificationSuite(spark) \ .onData(df) \ .addCheck( check.hasCompleteness('b',lambda x: x >= .5).where("b is not NULL") deequ support where clause pydeequ have not added the suppport for where clause

rahulgoyal2987 commented 3 years ago

where function can be added to checks.py which will call to the actual where function at line https://github.com/awslabs/deequ/blob/master/src/main/scala/com/amazon/deequ/checks/CheckWithLastConstraintFilterable.scala#L35