search

housepower / spark-clickhouse-connector

Spark ClickHouse Connector build on DataSourceV2 API
https://housepower.github.io/spark-clickhouse-connector
Apache License 2.0
171 stars 59 forks source link

Writing StructType to ClickHouse #278

Open MohsenIT opened 6 months ago

MohsenIT commented 6 months ago

I want to write a dataframe with struct type column into clickhouse. But it raises an error: Can't convert 'StructType([StructField('addon_id', StringType(), True), StructField('app_slug', StringType(), True), StructField('token', StringType(), True), StructField('uri', StringType(), True), StructField('in_browser', BooleanType(), True), StructField('addon_position', StringType(), True)])' datatype from spark to clickhouse.

  1. Why struct type does not convert to NestedType/Tuple/Json?
  2. How can we create NestedType/Tuple/ Json Column using spark-clickhouse-connector ?
pan3793 commented 6 months ago

The current implementation has very limited support for complex data types.