Closed Veiasai closed 3 months ago
encounter this when I use pyspark (3.5.0)
builder = ( SparkSession.builder.appName("local_spark") .master("local[*]") .config("spark.sql.catalog.clickhouse", "xenon.clickhouse.ClickHouseCatalog") .config( "spark.sql.catalog.clickhouse.host", host, ) .config("spark.sql.catalog.clickhouse.protocol", "http") .config("spark.sql.catalog.clickhouse.http_port", port) .config("spark.sql.catalog.clickhouse.user", user) .config("spark.sql.catalog.clickhouse.password", password) .config("spark.sql.catalog.clickhouse.database", db) .config("spark.sql.execution.arrow.pyspark.enabled", "true") .config("spark.sql.files.maxRecordsPerFile", "1000000") .config("spark.sql.execution.arrow.maxRecordsPerBatch", "100000") .config("spark.sql.session.timeZone", "UTC") .config( "spark.jars.repositories", "https://mirrors.sjtug.sjtu.edu.cn/maven-central/", ) .config( "spark.jars.packages", "com.github.housepower:clickhouse-spark-runtime-3.4_2.12:0.7.3,com.clickhouse:clickhouse-jdbc:0.4.6" ) )
@Veiasai what error are you getting?
Caused by version mismatch. clickhouse-spark-runtime-3.4_2.12 is only applicable to Spark 3.4 with Scala 2.12
clickhouse-spark-runtime-3.4_2.12
encounter this when I use pyspark (3.5.0)