default suggestion check error in iscontained

[poudelankit]

code for ConstraintSuggestion Default suggestions = ConstraintSuggestionRunner(spark).onData(df).addConstraintRule(DEFAULT()).run() for suggestion in suggestions['constraint_suggestions']: print(suggestion['code_for_constraint'])

Result of default suggestion The default constraint suggestion will return the following as a value under 'code_for_constraint': .isContainedIn("Category #1", ["Dental Surgery", "Laboratory"], lambda x: x >= 0.98, "It should be above 0.98!") which when applied to addCheck would result in error.

Check Implementation: check = Check(spark,CheckLevel.Error,"Manual Check") verification_runner = VerificationSuite(spark).onData(df).addCheck(check.isContainedIn("Category #1", ["Dental Surgery", "Laboratory"], lambda x: x >= 0.98, "It should be above 0.98!")) verification_result = verification_runner.run() df_checked = VerificationResult.checkResultsAsDataFrame(spark,verification_result) df_checked.show(truncate=False)

error TypeError: isContainedIn() takes 3 positional arguments but 5 were given

[chenliu0831]

This should fix the error https://github.com/awslabs/python-deequ/pull/157?

[poudelankit]

Implementation of scala in python worked for me::

Change was made on checks.py

def isContainedIn(self, column,allowed_values, assertion=None, hint=None): """

• Asserts that every non-null value in a column is contained in a set of predefined values *
  • @param column Column to run the assertion on
  • @param allowedValues Allowed values for the column
  • @param assertion Function that receives a double input parameter and returns a boolean
  • @param hint A hint to provide additional context why a constraint could have failed
  • @return */ """ if(assertion): assertion_func = ( ScalaFunction1(self._spark_session.sparkContext._gateway, assertion) ) hint = self._jvm.scala.Option.apply(hint) arr = self._spark_session.sparkContext._gateway.new_array(self._jvm.java.lang.String, len(allowed_values)) for i in range(len(allowed_values)): arr[i] = allowed_values[i] if assertion: self._Check = self._Check.isContainedIn(column, arr,assertion_func, hint) else: self._Check = self._Check.isContainedIn(column, arr) return self