ClickHouse / spark-clickhouse-connector

Spark ClickHouse Connector build on DataSourceV2 API
https://clickhouse.com/docs/en/integrations/apache-spark
Apache License 2.0
187 stars 66 forks source link

[QA] Hive and Clickhouse. How to select and insert data from clickhouse into hive using spark thrift server? #314

Closed paf91 closed 5 months ago

paf91 commented 5 months ago

So basically I'm using spark thrift server to let users use pure sql to select data from hive + s3 storage using spark. How do I select data from clickhouse using clickhouse-connector and THEN insert into Hive table? like: SET spark.sql.catalog.clickhouse.user=someuser; SET spark.sql.catalog.clickhouse.user=somepassword; use clickhouse;

then SELECT * FROM .

INSERT INTO .?

pan3793 commented 5 months ago

Use the following Spark SQL syntax

INSERT INTO spark_catalog.<hive_db>.<hive_tbl>
SELECT ... FROM clickhouse.<ch_db>.<ch_tbl>