search

awslabs / python-deequ

Python API for Deequ
Apache License 2.0
691 stars 132 forks source link

Bump PySpark to 3.1.1 #31

Open rjurney opened 3 years ago

rjurney commented 3 years ago

Is your feature request related to a problem? Please describe.

I can't use PyDeequ with my project because it only supports Python 2.4.7 and my package won't build. I want to bump the version of pyspark to 3.1.1.

Describe the solution you'd like

I want to upgrade to the latest version of PySpark.

Describe alternatives you've considered

I don't see an alternative if the project is to remain relevant. 3.0 has been out for a long while.

rjurney commented 3 years ago

I am also working on deequ#310 so please don't close.

shansfolder commented 3 years ago

+1

I got the exact same issue: I'd like to use pydeequ on pyspark 3.1.1

epilif1017a commented 2 years ago

Hi team,

Any news on this and even on compatibility with spark 3.1.2? I see the PR opened for 3.1.1 for a while, but seems stuck since april.

We are considering usage of pydeequ intensively, and apparently it works in 3.1.2 according to our tests (even with the error message "Deequ is still not supported in spark version: 3.1.2") , but would be cool to rest ensure that it is fully compatible and that the pydeequ community is looking towards a stable strategy to keep the project up to pace with spark releases.

Looking forward to hear from you :) Thanks!