Open rjurney opened 3 years ago
+1
I got the exact same issue: I'd like to use pydeequ on pyspark 3.1.1
Hi team,
Any news on this and even on compatibility with spark 3.1.2? I see the PR opened for 3.1.1 for a while, but seems stuck since april.
We are considering usage of pydeequ intensively, and apparently it works in 3.1.2 according to our tests (even with the error message "Deequ is still not supported in spark version: 3.1.2") , but would be cool to rest ensure that it is fully compatible and that the pydeequ community is looking towards a stable strategy to keep the project up to pace with spark releases.
Looking forward to hear from you :) Thanks!
Is your feature request related to a problem? Please describe.
I can't use PyDeequ with my project because it only supports Python 2.4.7 and my package won't build. I want to bump the version of pyspark to 3.1.1.
Describe the solution you'd like
I want to upgrade to the latest version of PySpark.
Describe alternatives you've considered
I don't see an alternative if the project is to remain relevant. 3.0 has been out for a long while.