Open MatthieuBlais opened 3 years ago
Which Glue version are you on?
Hi!
I am having exactly the same issue, on satisfies
and every other has*
function. Not running on Glue however .
I am using this test dockerized zeppelin.
The error:
Setup details in the images below
@MatthieuBlais I am sorry for taking over this answer. I believe our problem is very likely the same and the solution should be the same as well :)
Describe the issue: While running the tutorial code on AWS Glue, I can't get the checks to pass when there is an assertion. It seems to be a type issue.
"Can't execute the assertion: java.lang.String cannot be cast to java.lang.Boolean!"
Environment: Glue Development Endpoint Spark Version: 2.4 Python Version: 3 Pydeequ version: 0.1.5 Deequ jar: 1.0.3
Code: Using the tutorial sample code:
df = spark.sparkContext.parallelize([ Row(a="foo", b=1, c=5), Row(a="bar", b=2, c=6), Row(a="baz", b=3, c=None)]).toDF()
check = Check(spark, CheckLevel.Warning, "Review Check")
checkResult = VerificationSuite(spark) \ .onData(df) \ .addCheck( check.hasSize(lambda x: x >= 3) \ .hasMin("b", lambda x: x == 0) \ .isComplete("c") \ .isUnique("a") \ .isContainedIn("a", ["foo", "bar", "baz"]) \ .isNonNegative("b")) \ .run()
checkResult_df = VerificationResult.checkResultsAsDataFrame(spark, checkResult) checkResult_df.show()
Output: +------------+-----------+------------+--------------------+-----------------+--------------------+ | check|check_level|check_status| constraint|constraint_status| constraint_message| +------------+-----------+------------+--------------------+-----------------+--------------------+ |Review Check| Warning| Warning|SizeConstraint(Si...| Failure|Can't execute the...| |Review Check| Warning| Warning|MinimumConstraint...| Failure|Can't execute the...| |Review Check| Warning| Warning|CompletenessConst...| Failure|Value: 0.66666666...| |Review Check| Warning| Warning|UniquenessConstra...| Success| | |Review Check| Warning| Warning|ComplianceConstra...| Success| | |Review Check| Warning| Warning|ComplianceConstra...| Success| | +------------+-----------+------------+--------------------+-----------------+--------------------+
Expanding the two first rows with failure:
Row(check='Review Check', check_level='Warning', check_status='Warning', constraint='SizeConstraint(Size(None))', constraint_status='Failure', constraint_message="Can't execute the assertion: java.lang.String cannot be cast to java.lang.Boolean!") Row(check='Review Check', check_level='Warning', check_status='Warning', constraint='MinimumConstraint(Minimum(b,None))', constraint_status='Failure', constraint_message="Can't execute the assertion: java.lang.String cannot be cast to java.lang.Boolean!")