Can't execute the assertion: Error while sending a command.!

[MarsSu0618]

Hi, everyone. I have encountered a problem about check. my code follow as:

def verify_size_threshold(spark, dataframe, min_threshold):
    check = Check(spark, CheckLevel.Warning, "Review Size Check")

    checkResult = VerificationSuite(spark) \
        .onData(dataframe) \
        .addCheck(
            check.hasSize(lambda x: x >= min_threshold)) \
        .run()

    checkResult_df = VerificationResult.checkResultsAsDataFrame(spark, checkResult)

    return checkResult_df, check

and i get some Failure message, but my dataframe size is larger than min_threshold. message follow as:

constraint_status":"Failure","constraint_message":"Can't execute the assertion: Error while sending a command.!"

Hope someone can answer.

[ml6cz]

I had something similar, I didn't get that message specifically. I had issues with anything that uses a lambda assertion as seen here: https://github.com/awslabs/python-deequ/issues/54 .

I did have one successful run using a Glue Job to run constraints that use lambda assertions, but currently I am getting failures (not sure if there is an error on my part or something within PyDeequ was being update). Let me know if you get this to work in Glue though!

[gkirubhakaran]

I have the exact error for the same check

checkResult = VerificationSuite(spark) \
        .onData(dataframe) \
        .addCheck(
            check.hasSize(lambda x: x >= 1)) \
        .run()

However, it occurs only on certain dataframes and certain occasions. Is there a fix for this? Did anyone else do a work around?

[ruypeter]

In my case the problem appears using "hasUniqueness" check but only for one case. I have a lot of hasUniqueness checks working 100% but there's one with this error...

[LorenzoCarta]

We have the same problem reported by @ruypeter. This error is completely randomic but it appears only for Uniqueness checks.

[chenliu0831]

This will be resolved in next release which will include https://github.com/awslabs/python-deequ/issues/169.