ClickHouse / spark-clickhouse-connector

Spark ClickHouse Connector build on DataSourceV2 API
https://clickhouse.com/docs/en/integrations/apache-spark
Apache License 2.0
184 stars 65 forks source link

Unable to load custom metric object for class `xenon.clickhouse.BlocksReadMetric` #309

Closed Veiasai closed 3 months ago

Veiasai commented 5 months ago

encounter this when I use pyspark (3.5.0)

        builder = (
            SparkSession.builder.appName("local_spark")
            .master("local[*]")
            .config("spark.sql.catalog.clickhouse", "xenon.clickhouse.ClickHouseCatalog")
            .config(
                "spark.sql.catalog.clickhouse.host", host,
            )
            .config("spark.sql.catalog.clickhouse.protocol", "http")
            .config("spark.sql.catalog.clickhouse.http_port", port)
            .config("spark.sql.catalog.clickhouse.user", user)
            .config("spark.sql.catalog.clickhouse.password", password)
            .config("spark.sql.catalog.clickhouse.database", db)
            .config("spark.sql.execution.arrow.pyspark.enabled", "true")
            .config("spark.sql.files.maxRecordsPerFile", "1000000")
            .config("spark.sql.execution.arrow.maxRecordsPerBatch", "100000")
            .config("spark.sql.session.timeZone", "UTC")
            .config(
                "spark.jars.repositories",
                "https://mirrors.sjtug.sjtu.edu.cn/maven-central/",
            )
            .config(
                "spark.jars.packages",
                "com.github.housepower:clickhouse-spark-runtime-3.4_2.12:0.7.3,com.clickhouse:clickhouse-jdbc:0.4.6"
            )
        )
mzitnik commented 3 months ago

@Veiasai what error are you getting?

pan3793 commented 3 months ago

Caused by version mismatch. clickhouse-spark-runtime-3.4_2.12 is only applicable to Spark 3.4 with Scala 2.12