-
**Is your feature request related to a problem? Please describe.**
Currently only Spark 3.2 + Hadoop 3.2 + Scala 2.12 is supported. But from the first view the spark/pyspark code looks like it should…
-
**Describe the bug**
I have a batch inference pipelines. Each of these pipeline uses pydeequ to validate metrics of predictions and id. One of the checks are also anomalyCheck which make sure that …
-
### Current Behaviour
# converts the data types of the columns in the DataFrame to more appropriate types,
# useful for improving the performance of calculations.
…
-
**Describe the bug**
A clear and concise description of what the bug is.
It seems pydeequ is not working on the databricks runtime 14.0, whats the plan
**To Reproduce**
Steps to reproduce the beh…
-
pydeequ to be compatible with spark 3.5 and databricks runtime 14.0 onwards
-
For the joss review, I need to validate the performance test:
```Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off …
-
spark version - 3.3.2
deequ jar loaded , 2.0.3-spark-3.3
PyDeequ installed using pip , version 1.2.0
I am using the below code
from pyspark.sql import SparkSession, Row import pydeequ spark = (Sp…
-
The Check.satisfies method appears to be incompatible with Deequ 2.0.4. That release of Deequ includes a new optional `columns` parameter for the Check.satisfies() function per [PR 478](https://githu…
-
**Describe the bug**
When passing a lambda function for an assertion for hasSize, hasMin, or hasMax, it results in a "Can't execute the assertion" error.
**To Reproduce**
I use below code
fir…
-
**py4j.protocol.Py4JJavaError: An error occurred while calling None.org.apache.spark.api.java.JavaSparkContext.
: java.lang.IllegalAccessError: class org.apache.spark.storage.StorageUtils$ (in unname…