eakmanrq / sqlframe

Turning PySpark Into a Universal DataFrame API
https://sqlframe.readthedocs.io/en/stable/
MIT License
191 stars 3 forks source link

fix: Snowflake to properly convert variant types #33

Closed eakmanrq closed 2 months ago

eakmanrq commented 2 months ago

Snowflake by default leaves variant types like arrays and dicts as strings instead of their python types. In order for this to work we had to disable arrow conversion on Snowflake. See this PR for more details and eventual support for arrow so we don't have to do this anymore: https://github.com/snowflakedb/snowflake-connector-python/issues/544

Related to: https://github.com/eakmanrq/sqlframe/issues/31

Video: https://www.loom.com/share/945f5ad460f3475bb3a8ccce37d52ec3?sid=c16413b1-2ced-47b3-9c6b-1ec1d5a2f647