search

awslabs / python-deequ

Python API for Deequ
Apache License 2.0
702 stars 132 forks source link

Support pyspark=3.0.1 #4

Closed asears closed 3 years ago

asears commented 3 years ago

The latest pyspark should be supported, perhaps with 2.x for backwards compatibility.

https://pypi.org/project/pyspark/

gucciwang commented 3 years ago

We can't support Spark 3.0.x until Deequ compiles to Scala 2.12 (which is required for Spark 3.0). This is an issue with Deequ as well. Notably, this person opening this issue had the same error as you.

asears commented 3 years ago

I'll close this duplicate as seems like a Maven central build/package publishing task for the Scala deequ library.