housepower / spark-clickhouse-connector

Spark ClickHouse Connector build on DataSourceV2 API
https://housepower.github.io/spark-clickhouse-connector
Apache License 2.0
171 stars 59 forks source link

Unable to load custom metric object for class `xenon.clickhouse.BlocksReadMetric` #309

Open Veiasai opened 3 weeks ago

Veiasai commented 3 weeks ago

encounter this when I use pyspark (3.5.0)

        builder = (
            SparkSession.builder.appName("local_spark")
            .master("local[*]")
            .config("spark.sql.catalog.clickhouse", "xenon.clickhouse.ClickHouseCatalog")
            .config(
                "spark.sql.catalog.clickhouse.host", host,
            )
            .config("spark.sql.catalog.clickhouse.protocol", "http")
            .config("spark.sql.catalog.clickhouse.http_port", port)
            .config("spark.sql.catalog.clickhouse.user", user)
            .config("spark.sql.catalog.clickhouse.password", password)
            .config("spark.sql.catalog.clickhouse.database", db)
            .config("spark.sql.execution.arrow.pyspark.enabled", "true")
            .config("spark.sql.files.maxRecordsPerFile", "1000000")
            .config("spark.sql.execution.arrow.maxRecordsPerBatch", "100000")
            .config("spark.sql.session.timeZone", "UTC")
            .config(
                "spark.jars.repositories",
                "https://mirrors.sjtug.sjtu.edu.cn/maven-central/",
            )
            .config(
                "spark.jars.packages",
                "com.github.housepower:clickhouse-spark-runtime-3.4_2.12:0.7.3,com.clickhouse:clickhouse-jdbc:0.4.6"
            )
        )