Open tbbream opened 11 months ago
This is because there is no Spark-specific dialect implementation for Clickhouse, so Spark does not know how to convert this type to ClickhouseJDBC-compatible one: https://github.com/apache/spark/blob/b41ea9162f4c8fbc4d04d28d6ab5cc0342b88cb0/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JdbcUtils.scala#L139-L167
Describe the bug
From spark using the
com.clickhouse.jdbc.driver
, cannot save a dataframe that has a map type despite the table having a map and can perform this operation in other engines/code bases.The following error gets raised:
Steps to reproduce
Spin up clickhouse in a container. Run included python pyspark script.
Expected behaviour
Being able to save a Map type from spark using the jdbc driver
Code example
Error log
Configuration
Environment
ClickHouse server
CREATE TABLE
statements for tables involved: