Support Deequ 2.0.0 (Spark 3.1) and 2.0.1 (Spark 3.2)

[lecardozo]

Related issues

• 31

• 70

• 90

Related PRs

• 96

• 75

• 32

Changes

In order to support latest deequ releases, a few adjustments were necessary:

• [x] Added deequ 2.0.0-spark-3.1 and 2.0.1-spark-3.2 to the list of supported deequ/Spark versions
• [x] Adapted CategoricalRangeRule and FractionalRangeRule to include extra arguments for deequ>=2.0.0.
• [x] Improved tests to ensure functionality for suggestions

Additionally, a few technical debts were addressed

• [x] Now using SparkContext to retrieve the currently installed Spark version (no need to set SPARK_VERSION environment variable)
• [x] Added a Github workflow to test the library with multiple Spark versions

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

[lecardozo]

Hey, @gucciwang!

Wanted to hear your thoughts on this PR. Let me know if there's anything else we could do to enable this support for Deequ>=2.x 😃

Thanks!

[RaptorRob]

Can't wait for the fix of #70

[albjoaov]

🥺

[chenliu0831]

Is there a test report (if not from CI) that we can review? Just to make sure everything passes.

[lecardozo]

@chenliu0831 You can check the reports of tests here while I'm addressing your suggestions. As soon as we merge it, I'd expect to see them running on awslabs/pydeequ. :)

[chenliu0831]

@chenliu0831 You can check the reports of tests here while I'm addressing your suggestions. As soon as we merge it, I'd expect to see them running on awslabs/pydeequ. :)

Thanks, please just post a final link again when you are ready, I will review & approve

[lecardozo]

Done! Here is the last run of the CI workflow for c410f72

[mycaule]

Hi are you going to release the python package soon ?

[matbmeijer]

Hi, I would love to use this library. Does this PR support for spark 3.2.1, databricks runtime 10.4 LTS?