search

awslabs / python-deequ

Python API for Deequ
Apache License 2.0
669 stars 131 forks source link

AWS Glue Job runs indefinitely after exception occurs when adding Checks to the VerificationSuite #156

Open mattwaismann opened 9 months ago

mattwaismann commented 9 months ago

Describe the bug When running PyDeequ Checks on an AWS Glue job, if an exception occurs (e.g. accidentally calling hasCompletenss(...) instead of hasCompleteness(...)), the Glue job will keep running indefinitely.

To Reproduce Glue 4.0 deequ-2.0.4-spark-3.3.jar python-deequ 1.1.0

check_result = (
        VerificationSuite(spark)
        .onData(df)
        .addCheck(
            check.isComplete("primary_key")
            .hasCompletenss("product_name", lambda x: x > 0.999) #notice the typo
        )
        .run()

Expected behavior The expectation is that when errors such as this occur, the Glue job stops running or an exception is thrown so the callback server and spark session can be shut down:

Screenshots In the CloudWatch Glue error logs you see that the job continues polling for executor status:

image
mattwaismann commented 9 months ago

I experienced this issue again when I missed a positional argument when incorrectly calling check.hasCompleteness("col_name")

Expected behavior: Glue job stops with Exception: TypeError: hasCompleteness() missing 1 required positional argument: 'assertion'

Actual behavior: Glue job continued to run indefinitely