search

Nike-Inc / spark-expectations

A Python Library to support running data quality rules while the spark job is running⚡
https://engineering.nike.com/spark-expectations
Apache License 2.0
148 stars 32 forks source link

[BUG] Drop TempView is not supported latest Databricks clusters #84

Closed asingamaneni closed 2 months ago

asingamaneni commented 3 months ago

Describe the bug Getting the below error while running in JobCluster in shared mode of Databricks cluster in latest versions

py4j.security.Py4JSecurityException: Method public boolean org.apache.spark.sql.internal.CatalogImpl.dropTempView(java.lang.String) is not whitelisted on class class org.apache.spark.sql.internal.CatalogImpl

To Reproduce Steps to reproduce the behavior:

Expected behavior The job should not fail

Screenshots NA

Additional context NA