"TypeError: 'JavaPackage' object is not callable" while running any Deequ components

[alinesacchetti]

Describe the bug Currently our organization is trying to use PyDeequ libraries along with the Databricks which is using Apache Spark 3.3.2. When we try to call any function from pydeequ (AnalysisRunner, ColumnProfilerRunner, ConstraintSuggestionRunner, Check) we get the error "TypeError: 'JavaPackage' object is not callable"

To Reproduce Steps to reproduce the behavior:

1. Databricks cluster: 12.2 LTS (includes Apache Spark 3.3.2, Scala 2.12) using env vars:

• PYSPARK_PYTHON=/databricks/python3/bin/python3
• SPARK_VERSION=3.3.2

and spark conf:

spark.driver.extraJavaOptions "-Dlog4j2.formatMsgNoLookups=true" spark.databricks.optimizer.adaptive.enabled true spark.databricks.delta.preview.enabled true spark.sql.adaptive.coalescePartitions.enabled true spark.sql.sources.partitionOverwriteMode dynamic spark.sql.adaptive.skewJoin.enabled true spark.databricks.unity.catalog.enable false spark.sql.execution.arrow.enabled true spark.executor.extraJavaOptions "-Dlog4j2.formatMsgNoLookups=true"

3. Install pydeequ==1.2.0
4. Set up a PySpark session (https://pydeequ.readthedocs.io/en/latest/README.html#set-up-a-pyspark-session)
5. See error when you try to run any component (for example: https://pydeequ.readthedocs.io/en/latest/README.html#analyzers)

Expected behavior We hope that we can validate our data by cheking the last version of our data in an pyspark dataframe

Screenshots

Desktop (please complete the following information):

• OS: none
• Browser: chrome
• Version: 120.0.6099.217 64 bits

[chenliu0831]

Try this workaround - https://github.com/awslabs/python-deequ/issues/138#issuecomment-1611575546?

[alinesacchetti]

Hi! I tried to use this commands, but still have the same issue:

The weird part is that it doesn't matter which component I try to use, the error is always the same: "TypeError: 'JavaPackage' object is not callable"

[chenliu0831]

On Spark clusters, you probably will have better luck putting the Deequ jar to the Spark runtime jars library path / class path. We don't have a DB environment but you could probably follow this post https://aws.amazon.com/blogs/big-data/monitor-data-quality-in-your-data-lake-using-pydeequ-and-aws-glue/. You can download the Deequ jar from https://mvnrepository.com/artifact/com.amazon.deequ/deequ/2.0.4-spark-3.3

[chenliu0831]

Closing - feel free to re-open if you need more help.