search

great-expectations / great_expectations

Always know what to expect from your data.
https://docs.greatexpectations.io/
Apache License 2.0
9.71k stars 1.5k forks source link

Databricks Error - py4j.security.Py4JSecurityException: Constructor public org.apache.spark.SparkConf(boolean) is not whitelisted #7555

Open kujaska opened 1 year ago

kujaska commented 1 year ago

Details - https://github.com/great-expectations/great_expectations/issues/3538

While this issue is not reproducible on Databricks Community 11.3 LTS (includes Apache Spark 3.3.0, Scala 2.12), it is reproducible on AWS Databricks 12.2 LTS Shared cluster (includes Apache Spark 3.3.2, Scala 2.12) with great_expectations-0.16.5-py3-none-any.whl

Many thanks to @dbeswick-bupa - monkey-patch from https://github.com/great-expectations/great_expectations/issues/3538 works!

Please consider merging the monkey-patch to core library...

rdodev commented 1 year ago

Hey @kujaska thanks for bringing this up. I'll make engineering aware of the issue and monkey patch.