Available version of Scala, Spark, deequ for the latest pydeequ

yohei1126 commented 3 years ago

This is not a bug or feature report.

I am going to use pydeequ on Glue Notebook instance.

What is the latest version of pydeequ? 0.1.8? or 1.0.1?
- I can see 1.0.1 on pip repo. https://pypi.org/project/pydeequ/#history
- But I can not see it on Github repo.
What are available version of Scala, Spark, deequ for the latest version of pydeequ?

yohei1126 commented 3 years ago

Glue 1.0
Spark 2.4.3
Scala 2.11
Python 3.6
Deequ 1.1.0_spark-2.4-scala-2.11
PyDeequ 1.0.1

This combination works.

import pydeequ
from pydeequ.repository import *
pydeequ.__version__
--
Starting Spark application
1.0.1

yohei1126 commented 3 years ago

This combination did not work

Glue 3.0
Spark 3.1.1
Scala 2.12
Python 3.7
Deequ 2.0.0-spark-3.1
PyDeequ 1.0.1

import pydeequ
from pydeequ.repository import *
pydeequ.__version__
---
The code failed because of a fatal error:
    Session 0 unexpectedly reached final status 'dead'. See logs:
stdout: 

stderr: 
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/share/aws/glue/etl/jars/glue-assembly.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/lib/spark/jars/slf4j-log4j12-1.7.16.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]

dennisobrien commented 3 years ago

This issue https://github.com/awslabs/deequ/issues/372 was just closed 8 days ago so my guess is that a new release of pydeequ that supports Spark 3.1.x will be coming soon.

Tom-Hudson commented 2 years ago

When is 3.1 support coming?

awslabs / python-deequ

Available version of Scala, Spark, deequ for the latest pydeequ #74