search

MrPowers / quinn

pyspark methods to enhance developer productivity 📣 👯 🎉
https://mrpowers.github.io/quinn/
Apache License 2.0
597 stars 93 forks source link

0.10.1 causing issues on Databricks Unity Catalog clusters #190

Closed justingerolami closed 4 months ago

justingerolami commented 5 months ago

Started to have issues with Quinn crashing my code. Error is due to Unity Catalog whitelisted functions. Using pip to install quinn.

pip install quinn or pip install quinn == 0.10.1 cause the error. Usingpip install quinn == 0.9.0 does not cause any errors. 

py4j.security.Py4JSecurityException: Constructor public org.apache.spark.SparkConf(boolean) is not whitelisted.
29 if sys.modules["pyspark"].__version__ < "3.3.0":
     30     return df.select(col_name).rdd.flatMap(lambda x: x).collect()
---> 32 spark_config = df.sparkSession.sparkContext.getConf().getAll()
jeffbrennan commented 5 months ago

@justingerolami thanks for bringing this to our attention - I'll work on a fix today

MrPowers commented 5 months ago

@justingerolami - thanks for reporting this.

We are trying to work through all the issues and then will push another release.

MrPowers commented 5 months ago

@justingerolami - can you try v0.10.2 and let us know if this fixes your issue? https://github.com/MrPowers/quinn/releases/tag/v0.10.2

justingerolami commented 5 months ago

@MrPowers 0.10.2 seems to fix the issue! Thanks

MrPowers commented 5 months ago

Thanks for confirming @justingerolami

@jeffbrennan @SemyonSinchenko - do you think we should remove the 0.10.1 release from PyPI?

SemyonSinchenko commented 5 months ago

Thanks for confirming @justingerolami

@jeffbrennan @SemyonSinchenko - do you think we should remove the 0.10.1 release from PyPI?

I don't think it is a good idea. You may silently brake CI pipelines of people that already add this version into poetry.lock or requirements.txt files.

MrPowers commented 4 months ago

Good call, glad this is fixed now.