search

awslabs / python-deequ

Python API for Deequ
Apache License 2.0
691 stars 132 forks source link

using custom assertion on checks acts weird #46

Closed ankiiitraj closed 3 years ago

ankiiitraj commented 3 years ago

Example: check.hasDataType(parameter, ConstrainableDataTypes.Null, lambda x: True)

after checking gets done but program doesn't finishes, it keeps on running and when I force close the program, this is thrown

Exception ignored in: <module 'threading' from '/Users/araj7/.pyenv/versions/3.7.9/lib/python3.7/threading.py'> Traceback (most recent call last): File "/Users/araj7/.pyenv/versions/3.7.9/lib/python3.7/threading.py", line 1307, in _shutdown lock.acquire() File "/Users/araj7/Downloads/new/venv/lib/python3.7/site-packages/pyspark/context.py", line 269, in signal_handler self.cancelAllJobs() File "/Users/araj7/Downloads/new/venv/lib/python3.7/site-packages/pyspark/context.py", line 1039, in cancelAllJobs self._jsc.sc().cancelAllJobs() AttributeError: 'NoneType' object has no attribute 'sc'

PS: This doesn't happen if I don't use custom assertion.

ankiiitraj commented 3 years ago

Update, I ran this on Gule 2.0 and the glue job was continuously running and don't dies.

Whenever this is a custom assertion added this shows up on console. Python callback server has started

ankiiitraj commented 3 years ago

This issue has been solved in #7 Closing this.