search

awslabs / python-deequ

Python API for Deequ
Apache License 2.0
676 stars 131 forks source link

Adding Spark 3.1 #96

Closed chethanuk closed 1 year ago

chethanuk commented 2 years ago

Issue #, if available: Adding Support for Spark 3.1

Description of changes:

NOTE:

6 tests are failing, will look into it

E                   py4j.protocol.Py4JError: An error occurred while calling None.com.amazon.deequ.suggestions.rules.CategoricalRangeRule. Trace:
E                   py4j.Py4JException: Constructor com.amazon.deequ.suggestions.rules.CategoricalRangeRule([]) does not exist

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

chenliu0831 commented 1 year ago

Thanks for your contribution - I would prefer we use https://github.com/awslabs/python-deequ/pull/100 which seems more complete. Closing this for now.