eakmanrq / sqlframe

Turning PySpark Into a Universal DataFrame API
https://sqlframe.readthedocs.io/en/stable/
MIT License
290 stars 9 forks source link

fix: ignore conf if passed in config #61

Closed eakmanrq closed 4 months ago

eakmanrq commented 4 months ago

Potentially the more robust solution would be to parse the SparkConf object and check if it contains any SQLFrame related config but the documentation doesn't say this is supported and I wouldn't be surprised if not one does that so just ignoring for now.

For Spark engine the conf object is passed through (method is overriden).