search

ClickHouse / clickhouse-java

ClickHouse Java Clients & JDBC Driver
https://clickhouse.com
Apache License 2.0
1.45k stars 535 forks source link

[Bug] Always thrown `StreamingQueryException: No client available` when shaded jar is assembled with spark 2.4.5 (JDK1.8) #1227

Open oliverdding opened 1 year ago

oliverdding commented 1 year ago

I'm testing clickhouse-jdbc with Spark 2.4.5, but org.apache.spark.sql.streaming.StreamingQueryException: No client available is always thrown.

I swear that I added the classifier "all" like this(sbt):

"com.clickhouse" % "clickhouse-jdbc" % "0.4.0" classifier "all"

After jar -xf the assembled jar file, I could find the http client related class file:

image

Then I looked into the source code of clickhouse-http-client, and found this:

    CONNECTION_PROVIDER("http_connection_provider", HttpConnectionProvider.HTTP_URL_CONNECTION,
            "HTTP connection provider. HTTP_CLIENT is only supported in JDK 11 or above."),

I'm not sure if this is related, so I'm asking for help. Thanks for any advice.

zhicwu commented 1 year ago

The shaded jar does not contain JDK http client, so it should work under JDK 8 - we still have CI job to check that. Is it working fine before upgrading to 0.4.0? Have you removed the old driver(0.3.2*) from your classpath?

oliverdding commented 1 year ago

I haven't tried 0.3.2* yet, everything above is all about 0.4.0.

Below is part of the project dependencies:

Screenshot 2023-02-06 at 21 16 11

Following your advice, I tried 0.3.2-patch11 but it did not work:

image 
23/02/06 13:19:02 ERROR MicroBatchExecution: Query test.clickHouse [id = 57dc7540-0052-48b0-8d44-2a4a8a0b3db9, runId = 9d90a833-f0c3-4423-b69e-696745592086] terminated with error
java.sql.SQLException: No client available, server ClickHouseNode [uri=http://{HOST}:8123/test]@2085039148
        at com.clickhouse.jdbc.SqlExceptionUtils.handle(SqlExceptionUtils.java:85)
        at com.clickhouse.jdbc.SqlExceptionUtils.create(SqlExceptionUtils.java:31)
        at com.clickhouse.jdbc.SqlExceptionUtils.handle(SqlExceptionUtils.java:90)
        at com.clickhouse.jdbc.internal.ClickHouseConnectionImpl.getServerInfo(ClickHouseConnectionImpl.java:93)
        at com.clickhouse.jdbc.internal.ClickHouseConnectionImpl.<init>(ClickHouseConnectionImpl.java:298)
        at com.clickhouse.jdbc.internal.ClickHouseConnectionImpl.<init>(ClickHouseConnectionImpl.java:241)
        at com.clickhouse.jdbc.ClickHouseDriver.connect(ClickHouseDriver.java:145)
        at com.clickhouse.jdbc.ClickHouseDriver.connect(ClickHouseDriver.java:41)
        at org.apache.spark.sql.execution.datasources.jdbc.DriverWrapper.connect(DriverWrapper.scala:45)
        at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$$anonfun$createConnectionFactory$1.apply(JdbcUtils.scala:63)
        at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$$anonfun$createConnectionFactory$1.apply(JdbcUtils.scala:54)
        at org.apache.spark.sql.execution.datasources.jdbc.JdbcRelationProvider.createRelation(JdbcRelationProvider.scala:48)
        at org.apache.spark.sql.execution.datasources.SaveIntoDataSourceCommand.run(SaveIntoDataSourceCommand.scala:45)
        at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:70)
        at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:68)
        at org.apache.spark.sql.execution.command.ExecutedCommandExec.doExecute(commands.scala:86)
        at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:131)
        at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:127)
        at org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:155)
        at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
        at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:152)
        at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:127)
        at org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:83)
        at org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:81)
        at org.apache.spark.sql.DataFrameWriter$$anonfun$runCommand$1.apply(DataFrameWriter.scala:676)
        at org.apache.spark.sql.DataFrameWriter$$anonfun$runCommand$1.apply(DataFrameWriter.scala:676)
        at org.apache.spark.sql.execution.SQLExecution$$anonfun$withNewExecutionId$1.apply(SQLExecution.scala:80)
        at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:127)
        at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:75)
        at org.apache.spark.sql.DataFrameWriter.runCommand(DataFrameWriter.scala:676)
        at org.apache.spark.sql.DataFrameWriter.saveToV1Source(DataFrameWriter.scala:285)
        at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:271)
        at org.apache.spark.sql.DataFrameWriter.jdbc(DataFrameWriter.scala:515)
        at com.github.oliverdding.spark.test.jobs.OfficialClickHouseJdbc$$anonfun$3.apply(OfficialClickHouseJdbc.scala:44)
        at com.github.oliverdding.spark.test.jobs.OfficialClickHouseJdbc$$anonfun$3.apply(OfficialClickHouseJdbc.scala:28)
        at org.apache.spark.sql.execution.streaming.sources.ForeachBatchSink.addBatch(ForeachBatchSink.scala:35)
        at org.apache.spark.sql.execution.streaming.MicroBatchExecution$$anonfun$org$apache$spark$sql$execution$streaming$MicroBatchExecution$$runBatch$5$$anonfun$apply$17.apply(MicroBatchExecution.scala:537)
        at org.apache.spark.sql.execution.SQLExecution$$anonfun$withNewExecutionId$1.apply(SQLExecution.scala:80)
        at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:127)
        at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:75)
        at org.apache.spark.sql.execution.streaming.MicroBatchExecution$$anonfun$org$apache$spark$sql$execution$streaming$MicroBatchExecution$$runBatch$5.apply(MicroBatchExecution.scala:535)
        at org.apache.spark.sql.execution.streaming.ProgressReporter$class.reportTimeTaken(ProgressReporter.scala:351)
        at org.apache.spark.sql.execution.streaming.StreamExecution.reportTimeTaken(StreamExecution.scala:58)
        at org.apache.spark.sql.execution.streaming.MicroBatchExecution.org$apache$spark$sql$execution$streaming$MicroBatchExecution$$runBatch(MicroBatchExecution.scala:534)
        at org.apache.spark.sql.execution.streaming.MicroBatchExecution$$anonfun$runActivatedStream$1$$anonfun$apply$mcZ$sp$1.apply$mcV$sp(MicroBatchExecution.scala:198)
        at org.apache.spark.sql.execution.streaming.MicroBatchExecution$$anonfun$runActivatedStream$1$$anonfun$apply$mcZ$sp$1.apply(MicroBatchExecution.scala:166)
        at org.apache.spark.sql.execution.streaming.MicroBatchExecution$$anonfun$runActivatedStream$1$$anonfun$apply$mcZ$sp$1.apply(MicroBatchExecution.scala:166)
        at org.apache.spark.sql.execution.streaming.ProgressReporter$class.reportTimeTaken(ProgressReporter.scala:351)
        at org.apache.spark.sql.execution.streaming.StreamExecution.reportTimeTaken(StreamExecution.scala:58)
        at org.apache.spark.sql.execution.streaming.MicroBatchExecution$$anonfun$runActivatedStream$1.apply$mcZ$sp(MicroBatchExecution.scala:166)
        at org.apache.spark.sql.execution.streaming.ProcessingTimeExecutor.execute(TriggerExecutor.scala:56)
        at org.apache.spark.sql.execution.streaming.MicroBatchExecution.runActivatedStream(MicroBatchExecution.scala:160)
        at org.apache.spark.sql.execution.streaming.StreamExecution.org$apache$spark$sql$execution$streaming$StreamExecution$$runStream(StreamExecution.scala:281)
        at org.apache.spark.sql.execution.streaming.StreamExecution$$anon$1.run(StreamExecution.scala:193)
Caused by: java.net.ConnectException: No client available
        at com.clickhouse.client.ClickHouseClientBuilder$DummyClient.execute(ClickHouseClientBuilder.java:58)
        at com.clickhouse.client.ClickHouseClientBuilder$Agent.sendOnce(ClickHouseClientBuilder.java:284)
        at com.clickhouse.client.ClickHouseClientBuilder$Agent.send(ClickHouseClientBuilder.java:296)
        at com.clickhouse.client.ClickHouseClientBuilder$Agent.execute(ClickHouseClientBuilder.java:351)
        at com.clickhouse.client.ClickHouseClient.executeAndWait(ClickHouseClient.java:824)
        at com.clickhouse.client.ClickHouseRequest.executeAndWait(ClickHouseRequest.java:1972)
        at com.clickhouse.jdbc.internal.ClickHouseConnectionImpl.getServerInfo(ClickHouseConnectionImpl.java:90)
        ... 50 more
org.apache.spark.sql.streaming.StreamingQueryException: No client available, server ClickHouseNode [uri=http://{HOST}:8123/test]@2085039148
zhicwu commented 1 year ago

Could you add exclude("*", "*") for clickhouse-jdbc? It's to exclude transitive dependencies and only use the shaded jar.

oliverdding commented 1 year ago

Hi @zhicwu . I have added exclude("*", "*") like this:

    libraryDependencies ++= Seq(
      ...
      "com.clickhouse" % "clickhouse-jdbc" % "0.4.0" exclude("*", "*") classifier "all"
    )

But it doesn't seem to work yet.

23/02/07 05:47:38 ERROR MicroBatchExecution: Query test.clickHouse [id = 57dc7540-0052-48b0-8d44-2a4a8a0b3db9, runId = f50172c8-ccfa-4947-82f2-037ac98cc2a1] terminated with error
java.sql.SQLException: No client available, server ClickHouseNode [uri=http://{HOST}:8123/test]@-174080404
        at com.clickhouse.jdbc.SqlExceptionUtils.handle(SqlExceptionUtils.java:85)
        at com.clickhouse.jdbc.SqlExceptionUtils.create(SqlExceptionUtils.java:31)
        at com.clickhouse.jdbc.SqlExceptionUtils.handle(SqlExceptionUtils.java:90)
        at com.clickhouse.jdbc.internal.ClickHouseConnectionImpl.getServerInfo(ClickHouseConnectionImpl.java:124)
        at com.clickhouse.jdbc.internal.ClickHouseConnectionImpl.<init>(ClickHouseConnectionImpl.java:333)
        at com.clickhouse.jdbc.internal.ClickHouseConnectionImpl.<init>(ClickHouseConnectionImpl.java:274)
        at com.clickhouse.jdbc.ClickHouseDriver.connect(ClickHouseDriver.java:157)
        at com.clickhouse.jdbc.ClickHouseDriver.connect(ClickHouseDriver.java:41)
        at org.apache.spark.sql.execution.datasources.jdbc.DriverWrapper.connect(DriverWrapper.scala:45)
        at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$$anonfun$createConnectionFactory$1.apply(JdbcUtils.scala:63)
        at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$$anonfun$createConnectionFactory$1.apply(JdbcUtils.scala:54)
        at org.apache.spark.sql.execution.datasources.jdbc.JdbcRelationProvider.createRelation(JdbcRelationProvider.scala:48)
        at org.apache.spark.sql.execution.datasources.SaveIntoDataSourceCommand.run(SaveIntoDataSourceCommand.scala:45)
        at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:70)
        at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:68)
        at org.apache.spark.sql.execution.command.ExecutedCommandExec.doExecute(commands.scala:86)
        at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:131)
        at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:127)
        at org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:155)
        at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
        at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:152)
        at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:127)
        at org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:83)
        at org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:81)
        at org.apache.spark.sql.DataFrameWriter$$anonfun$runCommand$1.apply(DataFrameWriter.scala:676)
        at org.apache.spark.sql.DataFrameWriter$$anonfun$runCommand$1.apply(DataFrameWriter.scala:676)
        at org.apache.spark.sql.execution.SQLExecution$$anonfun$withNewExecutionId$1.apply(SQLExecution.scala:80)
        at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:127)
        at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:75)
        at org.apache.spark.sql.DataFrameWriter.runCommand(DataFrameWriter.scala:676)
        at org.apache.spark.sql.DataFrameWriter.saveToV1Source(DataFrameWriter.scala:285)
        at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:271)
        at org.apache.spark.sql.DataFrameWriter.jdbc(DataFrameWriter.scala:515)
        at com.github.oliverdding.spark.test.jobs.OfficialClickHouseJdbc$$anonfun$3.apply(OfficialClickHouseJdbc.scala:44)
        at com.github.oliverdding.spark.test.jobs.OfficialClickHouseJdbc$$anonfun$3.apply(OfficialClickHouseJdbc.scala:28)
        at org.apache.spark.sql.execution.streaming.sources.ForeachBatchSink.addBatch(ForeachBatchSink.scala:35)
        at org.apache.spark.sql.execution.streaming.MicroBatchExecution$$anonfun$org$apache$spark$sql$execution$streaming$MicroBatchExecution$$runBatch$5$$anonfun$apply$17.apply(MicroBatchExecution.scala:537)
        at org.apache.spark.sql.execution.SQLExecution$$anonfun$withNewExecutionId$1.apply(SQLExecution.scala:80)
        at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:127)
        at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:75)
        at org.apache.spark.sql.execution.streaming.MicroBatchExecution$$anonfun$org$apache$spark$sql$execution$streaming$MicroBatchExecution$$runBatch$5.apply(MicroBatchExecution.scala:535)
        at org.apache.spark.sql.execution.streaming.ProgressReporter$class.reportTimeTaken(ProgressReporter.scala:351)
        at org.apache.spark.sql.execution.streaming.StreamExecution.reportTimeTaken(StreamExecution.scala:58)
        at org.apache.spark.sql.execution.streaming.MicroBatchExecution.org$apache$spark$sql$execution$streaming$MicroBatchExecution$$runBatch(MicroBatchExecution.scala:534)
        at org.apache.spark.sql.execution.streaming.MicroBatchExecution$$anonfun$runActivatedStream$1$$anonfun$apply$mcZ$sp$1.apply$mcV$sp(MicroBatchExecution.scala:198)
        at org.apache.spark.sql.execution.streaming.MicroBatchExecution$$anonfun$runActivatedStream$1$$anonfun$apply$mcZ$sp$1.apply(MicroBatchExecution.scala:166)
        at org.apache.spark.sql.execution.streaming.MicroBatchExecution$$anonfun$runActivatedStream$1$$anonfun$apply$mcZ$sp$1.apply(MicroBatchExecution.scala:166)
        at org.apache.spark.sql.execution.streaming.ProgressReporter$class.reportTimeTaken(ProgressReporter.scala:351)
        at org.apache.spark.sql.execution.streaming.StreamExecution.reportTimeTaken(StreamExecution.scala:58)
        at org.apache.spark.sql.execution.streaming.MicroBatchExecution$$anonfun$runActivatedStream$1.apply$mcZ$sp(MicroBatchExecution.scala:166)
        at org.apache.spark.sql.execution.streaming.ProcessingTimeExecutor.execute(TriggerExecutor.scala:56)
        at org.apache.spark.sql.execution.streaming.MicroBatchExecution.runActivatedStream(MicroBatchExecution.scala:160)
        at org.apache.spark.sql.execution.streaming.StreamExecution.org$apache$spark$sql$execution$streaming$StreamExecution$$runStream(StreamExecution.scala:281)
        at org.apache.spark.sql.execution.streaming.StreamExecution$$anon$1.run(StreamExecution.scala:193)
Caused by: java.net.ConnectException: No client available
        at com.clickhouse.client.ClickHouseClientBuilder$DummyClient.execute(ClickHouseClientBuilder.java:61)
        at com.clickhouse.client.ClickHouseClientBuilder$Agent.sendOnce(ClickHouseClientBuilder.java:287)
        at com.clickhouse.client.ClickHouseClientBuilder$Agent.send(ClickHouseClientBuilder.java:299)
        at com.clickhouse.client.ClickHouseClientBuilder$Agent.execute(ClickHouseClientBuilder.java:354)
        at com.clickhouse.client.ClickHouseClient.executeAndWait(ClickHouseClient.java:884)
        at com.clickhouse.client.ClickHouseRequest.executeAndWait(ClickHouseRequest.java:2068)
        at com.clickhouse.jdbc.internal.ClickHouseConnectionImpl.getServerInfo(ClickHouseConnectionImpl.java:121)
        ... 50 more
org.apache.spark.sql.streaming.StreamingQueryException: No client available, server ClickHouseNode [uri=http://{HOST}:8123/test]@-174080404

Additional information:

oliverdding commented 1 year ago

Hi @zhicwu , then I manually add clickhouse-http-client like below:

    libraryDependencies ++= Seq(
      ...
      "com.clickhouse" % "clickhouse-jdbc" % "0.4.0" exclude("*", "*"),
      "com.clickhouse" % "clickhouse-http-client" % "0.4.0"
    )

Although necessary dependencies are shown, it doesn't seem to work yet.

image

Any advice are appreciated.

zhicwu commented 1 year ago

You don't need to add clickhouse-http-client specifically as the shaded jar did that for you. Have you tried to validate in standalone mode? I don't have a similar environment but I'll try when I get home.

oliverdding commented 1 year ago

Hi @zhicwu , If I just use "com.clickhouse" % "clickhouse-jdbc" % "0.4.0" exclude("*", "*") classifier "all", there is no clickhouse-*-client related jar file in the dependencies panel of the project, which I think is not expected.

image

I create a test project for this issue: https://github.com/oliverdding/clickhouse-jdbc-test

I set up clickhouse, kafka with docker, then run sparkapp in local mode.

image

Below is my running report:

  1. ❌ "com.clickhouse" % "clickhouse-jdbc" % "0.4.0" exclude("*", "*") classifier "all" and run in local mode
image
  1. ❌ "com.clickhouse" % "clickhouse-jdbc" % "0.4.0" classifier "all" and run in local mode
image
shahsiddharth08 commented 1 year ago

I have the same error so FWIW, I tried to copy these jars to SPARK_HOME/jars and the app worked for me.

clickhouse-cli-client-0.3.2-patch11.jar
clickhouse-http-client-0.3.2-patch11.jar
clickhouse-jdbc-0.3.2-patch11.jar
clickhouse-client-0.3.2-patch11.jar
oliverdding commented 1 year ago

Hi @shahsiddharth08 , thanks for your reply. I tried it and it works!

THAT IS STRANGE ! 🤣

image image

I have put:

企业微信截图_63286c6f-540b-44fa-978c-f60f7b789217

into $SPARK_HOME/jars folder.