search

ClickHouse / spark-clickhouse-connector

Spark ClickHouse Connector build on DataSourceV2 API
https://clickhouse.com/docs/en/integrations/apache-spark
Apache License 2.0
187 stars 66 forks source link

[question] Is there an option on connector to manage/configure inserts to multiple partitions? #368

Open ShishkovSpace opened 1 week ago

ShishkovSpace commented 1 week ago

Hi,

In our process, we are using connector to transfer data from HDFS to Clickhouse via Spark. And this process overwrites a large number of partitions (there is not much data in each of them). The target table in Clickhouse is a Distributed table, under which there are ReplicatedMergeTree tables on 6 nodes. Lately, we are increasingly encountering a problem where tasks in Spark crash with an error

xenon.clickhouse.exception.CHServerException: [HTTP]user@1.clickhouse.host:8123}/clustername [1002] Error writing request body to server, server ClickHouseNode [uri=http://1.clickhouse.host:8123/clustername, options={socket_timeout=600000,socket_keepalive=true,session_timeout=1200}]@any_id
    at xenon.clickhouse.client.NodeClient.syncInsert(NodeClient.scala:121)
    at xenon.clickhouse.client.NodeClient.syncInsertOutputJSONEachRow(NodeClient.scala:86)
    at xenon.clickhouse.write.ClickHouseWriter.$anonfun$doFlush$1(ClickHouseWriter.scala:232)
    at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
    at scala.util.Try$.apply(Try.scala:213)
    at xenon.clickhouse.Utils$.retry(Utils.scala:99)
    at xenon.clickhouse.write.ClickHouseWriter.doFlush(ClickHouseWriter.scala:230)
    at xenon.clickhouse.write.ClickHouseWriter.flush(ClickHouseWriter.scala:220)
    at xenon.clickhouse.write.ClickHouseWriter.write(ClickHouseWriter.scala:192)
    at xenon.clickhouse.write.ClickHouseWriter.write(ClickHouseWriter.scala:39)
    at org.apache.spark.sql.execution.datasources.v2.DataWritingSparkTask$.$anonfun$run$1(WriteToDataSourceV2Exec.scala:442)
    at org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1538)
    at org.apache.spark.sql.execution.datasources.v2.DataWritingSparkTask$.run(WriteToDataSourceV2Exec.scala:480)
    at org.apache.spark.sql.execution.datasources.v2.V2TableWriteExec.$anonfun$writeWithV2$2(WriteToDataSourceV2Exec.scala:381)
    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
    at org.apache.spark.scheduler.Task.run(Task.scala:136)
    at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:548)
    at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1504)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:551)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

We are using the following connector configuration in SparkConf

.config("spark.task.maxFailures", "6")
.config("spark.sql.catalog.clickhouse", "xenon.clickhouse.ClickHouseCatalog")
.config("spark.sql.catalog.clickhouse.host", "host")
.config("spark.sql.catalog.clickhouse.protocol", "http")
.config("spark.sql.catalog.clickhouse.http_port", "8123")
.config("spark.sql.catalog.clickhouse.user", "user")
.config("spark.sql.catalog.clickhouse.password", "password")
.config("spark.sql.catalog.clickhouse.database", "database")
.config("spark.sql.catalog.clickhouse.option.socket_keepalive", "true")
.config("spark.sql.catalog.clickhouse.option.session_timeout", "1200")
.config("spark.sql.catalog.clickhouse.option.socket_timeout", "600000")
.config("spark.clickhouse.write.format", "arrow")
.config("spark.clickhouse.write.distributed.convertLocal", "false")
.config("spark.clickhouse.write.batchSize", "100000")
.config("spark.clickhouse.write.maxRetry", "5")
.config("spark.clickhouse.write.retryInterval", 60)
.config("spark.clickhouse.write.repartitionByPartition", "false")
.config("spark.clickhouse.write.distributed.useClusterNodes", "true")
.config("spark.clickhouse.ignoreUnsupportedTransform", "true")
.config("spark.clickhouse.read.format", "binary")
.config("spark.clickhouse.read.distributed.convertLocal", "false")

Spark version - 3.3.0 Connector version - 0.7.3 (jdbc - 0.4.6)

One method we tried to fix it was to play with the batch_size per insert. But it is still not always possible to achieve stability and reliability for jobs.

We didn't find any descriptions of options and connector configurations for cases like this. There is only a config at the source code which we were used for configuration - https://github.com/ClickHouse/spark-clickhouse-connector/blob/v0.7.3/spark-3.3/clickhouse-spark/src/main/scala/org/apache/spark/sql/clickhouse/ClickHouseSQLConf.scala Could you please guid if there are any connector options/configs through which we can manage the above scenarios? Or jdbc driver options which we could configure to achieve stability per inserts?

pan3793 commented 1 week ago

the error code 1002 means UNKNOWN_EXCEPTION, have you checked the clickhouse server's log? is there any clue for the error?

pan3793 commented 1 week ago

I'm not sure if you have awared this article, which introduces the generic idea of writing optimization. the key idea is

So, to achieve better performance, we should re-organize the DataFrame to fit ClickHouse data layout before inserting.

I noticed you have disabled spark.clickhouse.write.repartitionByPartition, then how did you organize your data before inserting? could you please paste your Spark's DataFrame DAG and the writing stage info?

ShishkovSpace commented 6 days ago

the error code 1002 means UNKNOWN_EXCEPTION, have you checked the clickhouse server's log? is there any clue for the error?

Thanks for fast reply. Yes, we have checked server logs and there were no any evident related error traces in it.

All i have is error trace from failed Spark tasks

logs

``` 18:49:16.403 [Executor task launch worker for task 30.1 in stage 7.0 (TID 891)] ERROR org.apache.spark.util.Utils - Aborting task xenon.clickhouse.exception.CHServerException: [HTTP]user@1.clickhouse.host:8123}/clustername [1002] Error writing request body to server, server ClickHouseNode [uri=http://1.clickhouse.host:8123/clustername, options={socket_timeout=600000,socket_keepalive=true,session_timeout=1200}]@251648836 at xenon.clickhouse.client.NodeClient.syncInsert(NodeClient.scala:121) ~[clickhouse-spark-runtime-3.3_2.12-0.7.3.jar:?] at xenon.clickhouse.client.NodeClient.syncInsertOutputJSONEachRow(NodeClient.scala:86) ~[clickhouse-spark-runtime-3.3_2.12-0.7.3.jar:?] at xenon.clickhouse.write.ClickHouseWriter.$anonfun$doFlush$1(ClickHouseWriter.scala:232) ~[clickhouse-spark-runtime-3.3_2.12-0.7.3.jar:?] at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23) ~[scala-library-2.12.15.jar:?] at scala.util.Try$.apply(Try.scala:213) ~[scala-library-2.12.15.jar:?] at xenon.clickhouse.Utils$.retry(Utils.scala:99) ~[clickhouse-spark-runtime-3.3_2.12-0.7.3.jar:?] at xenon.clickhouse.write.ClickHouseWriter.doFlush(ClickHouseWriter.scala:230) ~[clickhouse-spark-runtime-3.3_2.12-0.7.3.jar:?] at xenon.clickhouse.write.ClickHouseWriter.flush(ClickHouseWriter.scala:220) ~[clickhouse-spark-runtime-3.3_2.12-0.7.3.jar:?] at xenon.clickhouse.write.ClickHouseWriter.write(ClickHouseWriter.scala:192) ~[clickhouse-spark-runtime-3.3_2.12-0.7.3.jar:?] at xenon.clickhouse.write.ClickHouseWriter.write(ClickHouseWriter.scala:39) ~[clickhouse-spark-runtime-3.3_2.12-0.7.3.jar:?] at org.apache.spark.sql.execution.datasources.v2.DataWritingSparkTask$.$anonfun$run$1(WriteToDataSourceV2Exec.scala:442) ~[spark-sql_2.12-3.3.0.jar:3.3.0] at org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1538) ~[spark-core_2.12-3.3.0.jar:3.3.0] at org.apache.spark.sql.execution.datasources.v2.DataWritingSparkTask$.run(WriteToDataSourceV2Exec.scala:480) ~[spark-sql_2.12-3.3.0.jar:3.3.0] at org.apache.spark.sql.execution.datasources.v2.V2TableWriteExec.$anonfun$writeWithV2$2(WriteToDataSourceV2Exec.scala:381) ~[spark-sql_2.12-3.3.0.jar:3.3.0] at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90) ~[spark-core_2.12-3.3.0.jar:3.3.0] at org.apache.spark.scheduler.Task.run(Task.scala:136) ~[spark-core_2.12-3.3.0.jar:3.3.0] at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:548) ~[spark-core_2.12-3.3.0.jar:3.3.0] at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1504) ~[spark-core_2.12-3.3.0.jar:3.3.0] at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:551) ~[spark-core_2.12-3.3.0.jar:3.3.0] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[?:1.8.0_102] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[?:1.8.0_102] at java.lang.Thread.run(Thread.java:745) ~[?:1.8.0_102] Caused by: com.clickhouse.client.ClickHouseException: Error writing request body to server, server ClickHouseNode [uri=http://1.clickhouse.host:8123/clustername, options={socket_timeout=600000,socket_keepalive=true,session_timeout=1200}]@251648836 at com.clickhouse.client.ClickHouseException.of(ClickHouseException.java:168) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)] at com.clickhouse.client.AbstractClient.lambda$execute$0(AbstractClient.java:275) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)] at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590) ~[?:1.8.0_102] ... 3 more Caused by: java.io.IOException: Error writing request body to server at sun.net.www.protocol.http.HttpURLConnection$StreamingOutputStream.checkError(HttpURLConnection.java:3479) ~[?:1.8.0_102] at sun.net.www.protocol.http.HttpURLConnection$StreamingOutputStream.write(HttpURLConnection.java:3462) ~[?:1.8.0_102] at com.clickhouse.data.stream.Lz4OutputStream.flushBuffer(Lz4OutputStream.java:34) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)] at com.clickhouse.data.stream.AbstractByteArrayOutputStream.writeBuffer(AbstractByteArrayOutputStream.java:79) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)] at com.clickhouse.data.ClickHouseInputStream.pipe(ClickHouseInputStream.java:889) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)] at com.clickhouse.data.ClickHouseInputStream.pipe(ClickHouseInputStream.java:504) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)] at com.clickhouse.client.http.ClickHouseHttpConnection.postData(ClickHouseHttpConnection.java:318) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)] at com.clickhouse.client.http.HttpUrlConnectionImpl.post(HttpUrlConnectionImpl.java:225) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)] at com.clickhouse.client.http.ClickHouseHttpClient.send(ClickHouseHttpClient.java:124) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)] at com.clickhouse.client.AbstractClient.sendAsync(AbstractClient.java:161) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)] at com.clickhouse.client.AbstractClient.lambda$execute$0(AbstractClient.java:273) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)] at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590) ~[?:1.8.0_102] ... 3 more Suppressed: java.io.IOException: Error writing request body to server at sun.net.www.protocol.http.HttpURLConnection$StreamingOutputStream.checkError(HttpURLConnection.java:3479) ~[?:1.8.0_102] at sun.net.www.protocol.http.HttpURLConnection$StreamingOutputStream.write(HttpURLConnection.java:3462) ~[?:1.8.0_102] at com.clickhouse.data.stream.Lz4OutputStream.flushBuffer(Lz4OutputStream.java:34) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)] at com.clickhouse.data.stream.Lz4OutputStream.flush(Lz4OutputStream.java:82) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)] at com.clickhouse.data.ClickHouseOutputStream.close(ClickHouseOutputStream.java:273) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)] at com.clickhouse.data.stream.Lz4OutputStream.close(Lz4OutputStream.java:92) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)] at com.clickhouse.client.http.ClickHouseHttpConnection.postData(ClickHouseHttpConnection.java:266) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)] at com.clickhouse.client.http.HttpUrlConnectionImpl.post(HttpUrlConnectionImpl.java:225) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)] at com.clickhouse.client.http.ClickHouseHttpClient.send(ClickHouseHttpClient.java:124) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)] at com.clickhouse.client.AbstractClient.sendAsync(AbstractClient.java:161) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)] at com.clickhouse.client.AbstractClient.lambda$execute$0(AbstractClient.java:273) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)] at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590) ~[?:1.8.0_102] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[?:1.8.0_102] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[?:1.8.0_102] at java.lang.Thread.run(Thread.java:745) ~[?:1.8.0_102] 18:49:16.422 [Executor task launch worker for task 30.1 in stage 7.0 (TID 891)] ERROR org.apache.spark.sql.execution.datasources.v2.DataWritingSparkTask - Aborting commit for partition 30 (task 891, attempt 1, stage 7.0) 18:49:16.423 [Executor task launch worker for task 30.1 in stage 7.0 (TID 891)] ERROR org.apache.spark.sql.execution.datasources.v2.DataWritingSparkTask - Aborted commit for partition 30 (task 891, attempt 1, stage 7.0) 18:49:16.427 [Executor task launch worker for task 30.1 in stage 7.0 (TID 891)] ERROR org.apache.spark.executor.Executor - Exception in task 30.1 in stage 7.0 (TID 891) xenon.clickhouse.exception.CHServerException: [HTTP]user@1.clickhouse.host:8123}/clustername [1002] Error writing request body to server, server ClickHouseNode [uri=http://1.clickhouse.host:8123/clustername, options={socket_timeout=600000,socket_keepalive=true,session_timeout=1200}]@251648836 at xenon.clickhouse.client.NodeClient.syncInsert(NodeClient.scala:121) ~[clickhouse-spark-runtime-3.3_2.12-0.7.3.jar:?] at xenon.clickhouse.client.NodeClient.syncInsertOutputJSONEachRow(NodeClient.scala:86) ~[clickhouse-spark-runtime-3.3_2.12-0.7.3.jar:?] at xenon.clickhouse.write.ClickHouseWriter.$anonfun$doFlush$1(ClickHouseWriter.scala:232) ~[clickhouse-spark-runtime-3.3_2.12-0.7.3.jar:?] at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23) ~[scala-library-2.12.15.jar:?] at scala.util.Try$.apply(Try.scala:213) ~[scala-library-2.12.15.jar:?] at xenon.clickhouse.Utils$.retry(Utils.scala:99) ~[clickhouse-spark-runtime-3.3_2.12-0.7.3.jar:?] at xenon.clickhouse.write.ClickHouseWriter.doFlush(ClickHouseWriter.scala:230) ~[clickhouse-spark-runtime-3.3_2.12-0.7.3.jar:?] at xenon.clickhouse.write.ClickHouseWriter.flush(ClickHouseWriter.scala:220) ~[clickhouse-spark-runtime-3.3_2.12-0.7.3.jar:?] at xenon.clickhouse.write.ClickHouseWriter.write(ClickHouseWriter.scala:192) ~[clickhouse-spark-runtime-3.3_2.12-0.7.3.jar:?] at xenon.clickhouse.write.ClickHouseWriter.write(ClickHouseWriter.scala:39) ~[clickhouse-spark-runtime-3.3_2.12-0.7.3.jar:?] at org.apache.spark.sql.execution.datasources.v2.DataWritingSparkTask$.$anonfun$run$1(WriteToDataSourceV2Exec.scala:442) ~[spark-sql_2.12-3.3.0.jar:3.3.0] at org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1538) ~[spark-core_2.12-3.3.0.jar:3.3.0] at org.apache.spark.sql.execution.datasources.v2.DataWritingSparkTask$.run(WriteToDataSourceV2Exec.scala:480) ~[spark-sql_2.12-3.3.0.jar:3.3.0] at org.apache.spark.sql.execution.datasources.v2.V2TableWriteExec.$anonfun$writeWithV2$2(WriteToDataSourceV2Exec.scala:381) ~[spark-sql_2.12-3.3.0.jar:3.3.0] at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90) ~[spark-core_2.12-3.3.0.jar:3.3.0] at org.apache.spark.scheduler.Task.run(Task.scala:136) ~[spark-core_2.12-3.3.0.jar:3.3.0] at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:548) ~[spark-core_2.12-3.3.0.jar:3.3.0] at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1504) ~[spark-core_2.12-3.3.0.jar:3.3.0] at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:551) ~[spark-core_2.12-3.3.0.jar:3.3.0] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[?:1.8.0_102] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[?:1.8.0_102] at java.lang.Thread.run(Thread.java:745) ~[?:1.8.0_102] Caused by: com.clickhouse.client.ClickHouseException: Error writing request body to server, server ClickHouseNode [uri=http://1.clickhouse.host:8123/clustername, options={socket_timeout=600000,socket_keepalive=true,session_timeout=1200}]@251648836 at com.clickhouse.client.ClickHouseException.of(ClickHouseException.java:168) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)] at com.clickhouse.client.AbstractClient.lambda$execute$0(AbstractClient.java:275) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)] at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590) ~[?:1.8.0_102] ... 3 more Caused by: java.io.IOException: Error writing request body to server at sun.net.www.protocol.http.HttpURLConnection$StreamingOutputStream.checkError(HttpURLConnection.java:3479) ~[?:1.8.0_102] at sun.net.www.protocol.http.HttpURLConnection$StreamingOutputStream.write(HttpURLConnection.java:3462) ~[?:1.8.0_102] at com.clickhouse.data.stream.Lz4OutputStream.flushBuffer(Lz4OutputStream.java:34) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)] at com.clickhouse.data.stream.AbstractByteArrayOutputStream.writeBuffer(AbstractByteArrayOutputStream.java:79) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)] at com.clickhouse.data.ClickHouseInputStream.pipe(ClickHouseInputStream.java:889) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)] at com.clickhouse.data.ClickHouseInputStream.pipe(ClickHouseInputStream.java:504) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)] at com.clickhouse.client.http.ClickHouseHttpConnection.postData(ClickHouseHttpConnection.java:318) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)] at com.clickhouse.client.http.HttpUrlConnectionImpl.post(HttpUrlConnectionImpl.java:225) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)] at com.clickhouse.client.http.ClickHouseHttpClient.send(ClickHouseHttpClient.java:124) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)] at com.clickhouse.client.AbstractClient.sendAsync(AbstractClient.java:161) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)] at com.clickhouse.client.AbstractClient.lambda$execute$0(AbstractClient.java:273) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)] at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590) ~[?:1.8.0_102] ... 3 more Suppressed: java.io.IOException: Error writing request body to server at sun.net.www.protocol.http.HttpURLConnection$StreamingOutputStream.checkError(HttpURLConnection.java:3479) ~[?:1.8.0_102] at sun.net.www.protocol.http.HttpURLConnection$StreamingOutputStream.write(HttpURLConnection.java:3462) ~[?:1.8.0_102] at com.clickhouse.data.stream.Lz4OutputStream.flushBuffer(Lz4OutputStream.java:34) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)] at com.clickhouse.data.stream.Lz4OutputStream.flush(Lz4OutputStream.java:82) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)] at com.clickhouse.data.ClickHouseOutputStream.close(ClickHouseOutputStream.java:273) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)] at com.clickhouse.data.stream.Lz4OutputStream.close(Lz4OutputStream.java:92) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)] at com.clickhouse.client.http.ClickHouseHttpConnection.postData(ClickHouseHttpConnection.java:266) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)] at com.clickhouse.client.http.HttpUrlConnectionImpl.post(HttpUrlConnectionImpl.java:225) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)] at com.clickhouse.client.http.ClickHouseHttpClient.send(ClickHouseHttpClient.java:124) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)] at com.clickhouse.client.AbstractClient.sendAsync(AbstractClient.java:161) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)] at com.clickhouse.client.AbstractClient.lambda$execute$0(AbstractClient.java:273) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)] at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590) ~[?:1.8.0_102] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[?:1.8.0_102] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[?:1.8.0_102] at java.lang.Thread.run(Thread.java:745) ~[?:1.8.0_102] 18:49:17.657 [Executor task launch worker for task 5.4 in stage 7.0 (TID 890)] ERROR org.apache.spark.util.Utils - Aborting task xenon.clickhouse.exception.CHServerException: [HTTP]user@2.clickhouse.host:8123}/clustername [1002] Error writing request body to server, server ClickHouseNode [uri=http://2.clickhouse.host:8123/clustername, options={socket_timeout=600000,socket_keepalive=true,session_timeout=1200}]@-821203095 at xenon.clickhouse.client.NodeClient.syncInsert(NodeClient.scala:121) ~[clickhouse-spark-runtime-3.3_2.12-0.7.3.jar:?] at xenon.clickhouse.client.NodeClient.syncInsertOutputJSONEachRow(NodeClient.scala:86) ~[clickhouse-spark-runtime-3.3_2.12-0.7.3.jar:?] at xenon.clickhouse.write.ClickHouseWriter.$anonfun$doFlush$1(ClickHouseWriter.scala:232) ~[clickhouse-spark-runtime-3.3_2.12-0.7.3.jar:?] at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23) ~[scala-library-2.12.15.jar:?] at scala.util.Try$.apply(Try.scala:213) ~[scala-library-2.12.15.jar:?] at xenon.clickhouse.Utils$.retry(Utils.scala:99) ~[clickhouse-spark-runtime-3.3_2.12-0.7.3.jar:?] at xenon.clickhouse.write.ClickHouseWriter.doFlush(ClickHouseWriter.scala:230) ~[clickhouse-spark-runtime-3.3_2.12-0.7.3.jar:?] at xenon.clickhouse.write.ClickHouseWriter.flush(ClickHouseWriter.scala:220) ~[clickhouse-spark-runtime-3.3_2.12-0.7.3.jar:?] at xenon.clickhouse.write.ClickHouseWriter.write(ClickHouseWriter.scala:192) ~[clickhouse-spark-runtime-3.3_2.12-0.7.3.jar:?] at xenon.clickhouse.write.ClickHouseWriter.write(ClickHouseWriter.scala:39) ~[clickhouse-spark-runtime-3.3_2.12-0.7.3.jar:?] at org.apache.spark.sql.execution.datasources.v2.DataWritingSparkTask$.$anonfun$run$1(WriteToDataSourceV2Exec.scala:442) ~[spark-sql_2.12-3.3.0.jar:3.3.0] at org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1538) ~[spark-core_2.12-3.3.0.jar:3.3.0] at org.apache.spark.sql.execution.datasources.v2.DataWritingSparkTask$.run(WriteToDataSourceV2Exec.scala:480) ~[spark-sql_2.12-3.3.0.jar:3.3.0] at org.apache.spark.sql.execution.datasources.v2.V2TableWriteExec.$anonfun$writeWithV2$2(WriteToDataSourceV2Exec.scala:381) ~[spark-sql_2.12-3.3.0.jar:3.3.0] at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90) ~[spark-core_2.12-3.3.0.jar:3.3.0] at org.apache.spark.scheduler.Task.run(Task.scala:136) ~[spark-core_2.12-3.3.0.jar:3.3.0] at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:548) ~[spark-core_2.12-3.3.0.jar:3.3.0] at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1504) ~[spark-core_2.12-3.3.0.jar:3.3.0] at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:551) ~[spark-core_2.12-3.3.0.jar:3.3.0] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[?:1.8.0_102] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[?:1.8.0_102] at java.lang.Thread.run(Thread.java:745) ~[?:1.8.0_102] Caused by: com.clickhouse.client.ClickHouseException: Error writing request body to server, server ClickHouseNode [uri=http://2.clickhouse.host:8123/clustername, options={socket_timeout=600000,socket_keepalive=true,session_timeout=1200}]@-821203095 at com.clickhouse.client.ClickHouseException.of(ClickHouseException.java:168) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)] at com.clickhouse.client.AbstractClient.lambda$execute$0(AbstractClient.java:275) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)] at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590) ~[?:1.8.0_102] ... 3 more Caused by: java.io.IOException: Error writing request body to server at sun.net.www.protocol.http.HttpURLConnection$StreamingOutputStream.checkError(HttpURLConnection.java:3479) ~[?:1.8.0_102] at sun.net.www.protocol.http.HttpURLConnection$StreamingOutputStream.write(HttpURLConnection.java:3462) ~[?:1.8.0_102] at com.clickhouse.data.stream.Lz4OutputStream.flushBuffer(Lz4OutputStream.java:34) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)] at com.clickhouse.data.stream.AbstractByteArrayOutputStream.writeBuffer(AbstractByteArrayOutputStream.java:79) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)] at com.clickhouse.data.ClickHouseInputStream.pipe(ClickHouseInputStream.java:889) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)] at com.clickhouse.data.ClickHouseInputStream.pipe(ClickHouseInputStream.java:504) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)] at com.clickhouse.client.http.ClickHouseHttpConnection.postData(ClickHouseHttpConnection.java:318) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)] at com.clickhouse.client.http.HttpUrlConnectionImpl.post(HttpUrlConnectionImpl.java:225) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)] at com.clickhouse.client.http.ClickHouseHttpClient.send(ClickHouseHttpClient.java:124) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)] at com.clickhouse.client.AbstractClient.sendAsync(AbstractClient.java:161) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)] at com.clickhouse.client.AbstractClient.lambda$execute$0(AbstractClient.java:273) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)] at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590) ~[?:1.8.0_102] ... 3 more Suppressed: java.io.IOException: Error writing request body to server at sun.net.www.protocol.http.HttpURLConnection$StreamingOutputStream.checkError(HttpURLConnection.java:3479) ~[?:1.8.0_102] at sun.net.www.protocol.http.HttpURLConnection$StreamingOutputStream.write(HttpURLConnection.java:3462) ~[?:1.8.0_102] at com.clickhouse.data.stream.Lz4OutputStream.flushBuffer(Lz4OutputStream.java:34) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)] at com.clickhouse.data.stream.Lz4OutputStream.flush(Lz4OutputStream.java:82) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)] at com.clickhouse.data.ClickHouseOutputStream.close(ClickHouseOutputStream.java:273) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)] at com.clickhouse.data.stream.Lz4OutputStream.close(Lz4OutputStream.java:92) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)] at com.clickhouse.client.http.ClickHouseHttpConnection.postData(ClickHouseHttpConnection.java:266) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)] at com.clickhouse.client.http.HttpUrlConnectionImpl.post(HttpUrlConnectionImpl.java:225) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)] at com.clickhouse.client.http.ClickHouseHttpClient.send(ClickHouseHttpClient.java:124) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)] at com.clickhouse.client.AbstractClient.sendAsync(AbstractClient.java:161) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)] at com.clickhouse.client.AbstractClient.lambda$execute$0(AbstractClient.java:273) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)] at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590) ~[?:1.8.0_102] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[?:1.8.0_102] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[?:1.8.0_102] at java.lang.Thread.run(Thread.java:745) ~[?:1.8.0_102] 18:49:17.659 [Executor task launch worker for task 5.4 in stage 7.0 (TID 890)] ERROR org.apache.spark.sql.execution.datasources.v2.DataWritingSparkTask - Aborting commit for partition 5 (task 890, attempt 4, stage 7.0) 18:49:17.660 [Executor task launch worker for task 5.4 in stage 7.0 (TID 890)] ERROR org.apache.spark.sql.execution.datasources.v2.DataWritingSparkTask - Aborted commit for partition 5 (task 890, attempt 4, stage 7.0) 18:49:17.661 [Executor task launch worker for task 5.4 in stage 7.0 (TID 890)] ERROR org.apache.spark.executor.Executor - Exception in task 5.4 in stage 7.0 (TID 890) xenon.clickhouse.exception.CHServerException: [HTTP]user@2.clickhouse.host:8123}/clustername [1002] Error writing request body to server, server ClickHouseNode [uri=http://2.clickhouse.host:8123/clustername, options={socket_timeout=600000,socket_keepalive=true,session_timeout=1200}]@-821203095 at xenon.clickhouse.client.NodeClient.syncInsert(NodeClient.scala:121) ~[clickhouse-spark-runtime-3.3_2.12-0.7.3.jar:?] at xenon.clickhouse.client.NodeClient.syncInsertOutputJSONEachRow(NodeClient.scala:86) ~[clickhouse-spark-runtime-3.3_2.12-0.7.3.jar:?] at xenon.clickhouse.write.ClickHouseWriter.$anonfun$doFlush$1(ClickHouseWriter.scala:232) ~[clickhouse-spark-runtime-3.3_2.12-0.7.3.jar:?] at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23) ~[scala-library-2.12.15.jar:?] at scala.util.Try$.apply(Try.scala:213) ~[scala-library-2.12.15.jar:?] at xenon.clickhouse.Utils$.retry(Utils.scala:99) ~[clickhouse-spark-runtime-3.3_2.12-0.7.3.jar:?] at xenon.clickhouse.write.ClickHouseWriter.doFlush(ClickHouseWriter.scala:230) ~[clickhouse-spark-runtime-3.3_2.12-0.7.3.jar:?] at xenon.clickhouse.write.ClickHouseWriter.flush(ClickHouseWriter.scala:220) ~[clickhouse-spark-runtime-3.3_2.12-0.7.3.jar:?] at xenon.clickhouse.write.ClickHouseWriter.write(ClickHouseWriter.scala:192) ~[clickhouse-spark-runtime-3.3_2.12-0.7.3.jar:?] at xenon.clickhouse.write.ClickHouseWriter.write(ClickHouseWriter.scala:39) ~[clickhouse-spark-runtime-3.3_2.12-0.7.3.jar:?] at org.apache.spark.sql.execution.datasources.v2.DataWritingSparkTask$.$anonfun$run$1(WriteToDataSourceV2Exec.scala:442) ~[spark-sql_2.12-3.3.0.jar:3.3.0] at org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1538) ~[spark-core_2.12-3.3.0.jar:3.3.0] at org.apache.spark.sql.execution.datasources.v2.DataWritingSparkTask$.run(WriteToDataSourceV2Exec.scala:480) ~[spark-sql_2.12-3.3.0.jar:3.3.0] at org.apache.spark.sql.execution.datasources.v2.V2TableWriteExec.$anonfun$writeWithV2$2(WriteToDataSourceV2Exec.scala:381) ~[spark-sql_2.12-3.3.0.jar:3.3.0] at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90) ~[spark-core_2.12-3.3.0.jar:3.3.0] at org.apache.spark.scheduler.Task.run(Task.scala:136) ~[spark-core_2.12-3.3.0.jar:3.3.0] at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:548) ~[spark-core_2.12-3.3.0.jar:3.3.0] at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1504) ~[spark-core_2.12-3.3.0.jar:3.3.0] at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:551) ~[spark-core_2.12-3.3.0.jar:3.3.0] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[?:1.8.0_102] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[?:1.8.0_102] at java.lang.Thread.run(Thread.java:745) ~[?:1.8.0_102] Caused by: com.clickhouse.client.ClickHouseException: Error writing request body to server, server ClickHouseNode [uri=http://2.clickhouse.host:8123/clustername, options={socket_timeout=600000,socket_keepalive=true,session_timeout=1200}]@-821203095 at com.clickhouse.client.ClickHouseException.of(ClickHouseException.java:168) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)] at com.clickhouse.client.AbstractClient.lambda$execute$0(AbstractClient.java:275) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)] at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590) ~[?:1.8.0_102] ... 3 more Caused by: java.io.IOException: Error writing request body to server at sun.net.www.protocol.http.HttpURLConnection$StreamingOutputStream.checkError(HttpURLConnection.java:3479) ~[?:1.8.0_102] at sun.net.www.protocol.http.HttpURLConnection$StreamingOutputStream.write(HttpURLConnection.java:3462) ~[?:1.8.0_102] at com.clickhouse.data.stream.Lz4OutputStream.flushBuffer(Lz4OutputStream.java:34) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)] at com.clickhouse.data.stream.AbstractByteArrayOutputStream.writeBuffer(AbstractByteArrayOutputStream.java:79) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)] at com.clickhouse.data.ClickHouseInputStream.pipe(ClickHouseInputStream.java:889) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)] at com.clickhouse.data.ClickHouseInputStream.pipe(ClickHouseInputStream.java:504) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)] at com.clickhouse.client.http.ClickHouseHttpConnection.postData(ClickHouseHttpConnection.java:318) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)] at com.clickhouse.client.http.HttpUrlConnectionImpl.post(HttpUrlConnectionImpl.java:225) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)] at com.clickhouse.client.http.ClickHouseHttpClient.send(ClickHouseHttpClient.java:124) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)] at com.clickhouse.client.AbstractClient.sendAsync(AbstractClient.java:161) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)] at com.clickhouse.client.AbstractClient.lambda$execute$0(AbstractClient.java:273) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)] at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590) ~[?:1.8.0_102] ... 3 more Suppressed: java.io.IOException: Error writing request body to server at sun.net.www.protocol.http.HttpURLConnection$StreamingOutputStream.checkError(HttpURLConnection.java:3479) ~[?:1.8.0_102] at sun.net.www.protocol.http.HttpURLConnection$StreamingOutputStream.write(HttpURLConnection.java:3462) ~[?:1.8.0_102] at com.clickhouse.data.stream.Lz4OutputStream.flushBuffer(Lz4OutputStream.java:34) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)] at com.clickhouse.data.stream.Lz4OutputStream.flush(Lz4OutputStream.java:82) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)] at com.clickhouse.data.ClickHouseOutputStream.close(ClickHouseOutputStream.java:273) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)] at com.clickhouse.data.stream.Lz4OutputStream.close(Lz4OutputStream.java:92) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)] at com.clickhouse.client.http.ClickHouseHttpConnection.postData(ClickHouseHttpConnection.java:266) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)] at com.clickhouse.client.http.HttpUrlConnectionImpl.post(HttpUrlConnectionImpl.java:225) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)] at com.clickhouse.client.http.ClickHouseHttpClient.send(ClickHouseHttpClient.java:124) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)] at com.clickhouse.client.AbstractClient.sendAsync(AbstractClient.java:161) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)] at com.clickhouse.client.AbstractClient.lambda$execute$0(AbstractClient.java:273) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)] at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590) ~[?:1.8.0_102] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[?:1.8.0_102] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[?:1.8.0_102] at java.lang.Thread.run(Thread.java:745) ~[?:1.8.0_102] 18:49:17.812 [Executor task launch worker for task 0.4 in stage 7.0 (TID 887)] ERROR org.apache.spark.util.Utils - Aborting task xenon.clickhouse.exception.CHServerException: [HTTP]user@2.clickhouse.host:8123}/clustername [1002] Error writing request body to server, server ClickHouseNode [uri=http://2.clickhouse.host:8123/clustername, options={socket_timeout=600000,socket_keepalive=true,session_timeout=1200}]@-412378870 at xenon.clickhouse.client.NodeClient.syncInsert(NodeClient.scala:121) ~[clickhouse-spark-runtime-3.3_2.12-0.7.3.jar:?] at xenon.clickhouse.client.NodeClient.syncInsertOutputJSONEachRow(NodeClient.scala:86) ~[clickhouse-spark-runtime-3.3_2.12-0.7.3.jar:?] at xenon.clickhouse.write.ClickHouseWriter.$anonfun$doFlush$1(ClickHouseWriter.scala:232) ~[clickhouse-spark-runtime-3.3_2.12-0.7.3.jar:?] at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23) ~[scala-library-2.12.15.jar:?] at scala.util.Try$.apply(Try.scala:213) ~[scala-library-2.12.15.jar:?] at xenon.clickhouse.Utils$.retry(Utils.scala:99) ~[clickhouse-spark-runtime-3.3_2.12-0.7.3.jar:?] at xenon.clickhouse.write.ClickHouseWriter.doFlush(ClickHouseWriter.scala:230) ~[clickhouse-spark-runtime-3.3_2.12-0.7.3.jar:?] at xenon.clickhouse.write.ClickHouseWriter.flush(ClickHouseWriter.scala:220) ~[clickhouse-spark-runtime-3.3_2.12-0.7.3.jar:?] at xenon.clickhouse.write.ClickHouseWriter.write(ClickHouseWriter.scala:192) ~[clickhouse-spark-runtime-3.3_2.12-0.7.3.jar:?] at xenon.clickhouse.write.ClickHouseWriter.write(ClickHouseWriter.scala:39) ~[clickhouse-spark-runtime-3.3_2.12-0.7.3.jar:?] at org.apache.spark.sql.execution.datasources.v2.DataWritingSparkTask$.$anonfun$run$1(WriteToDataSourceV2Exec.scala:442) ~[spark-sql_2.12-3.3.0.jar:3.3.0] at org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1538) ~[spark-core_2.12-3.3.0.jar:3.3.0] at org.apache.spark.sql.execution.datasources.v2.DataWritingSparkTask$.run(WriteToDataSourceV2Exec.scala:480) ~[spark-sql_2.12-3.3.0.jar:3.3.0] at org.apache.spark.sql.execution.datasources.v2.V2TableWriteExec.$anonfun$writeWithV2$2(WriteToDataSourceV2Exec.scala:381) ~[spark-sql_2.12-3.3.0.jar:3.3.0] at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90) ~[spark-core_2.12-3.3.0.jar:3.3.0] at org.apache.spark.scheduler.Task.run(Task.scala:136) ~[spark-core_2.12-3.3.0.jar:3.3.0] at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:548) ~[spark-core_2.12-3.3.0.jar:3.3.0] at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1504) ~[spark-core_2.12-3.3.0.jar:3.3.0] at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:551) ~[spark-core_2.12-3.3.0.jar:3.3.0] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[?:1.8.0_102] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[?:1.8.0_102] at java.lang.Thread.run(Thread.java:745) ~[?:1.8.0_102] Caused by: com.clickhouse.client.ClickHouseException: Error writing request body to server, server ClickHouseNode [uri=http://2.clickhouse.host:8123/clustername, options={socket_timeout=600000,socket_keepalive=true,session_timeout=1200}]@-412378870 at com.clickhouse.client.ClickHouseException.of(ClickHouseException.java:168) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)] at com.clickhouse.client.AbstractClient.lambda$execute$0(AbstractClient.java:275) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)] at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590) ~[?:1.8.0_102] ... 3 more Caused by: java.io.IOException: Error writing request body to server at sun.net.www.protocol.http.HttpURLConnection$StreamingOutputStream.checkError(HttpURLConnection.java:3479) ~[?:1.8.0_102] at sun.net.www.protocol.http.HttpURLConnection$StreamingOutputStream.write(HttpURLConnection.java:3462) ~[?:1.8.0_102] at com.clickhouse.data.stream.Lz4OutputStream.flushBuffer(Lz4OutputStream.java:34) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)] at com.clickhouse.data.stream.AbstractByteArrayOutputStream.writeBuffer(AbstractByteArrayOutputStream.java:79) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)] at com.clickhouse.data.ClickHouseInputStream.pipe(ClickHouseInputStream.java:889) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)] at com.clickhouse.data.ClickHouseInputStream.pipe(ClickHouseInputStream.java:504) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)] at com.clickhouse.client.http.ClickHouseHttpConnection.postData(ClickHouseHttpConnection.java:318) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)] at com.clickhouse.client.http.HttpUrlConnectionImpl.post(HttpUrlConnectionImpl.java:225) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)] at com.clickhouse.client.http.ClickHouseHttpClient.send(ClickHouseHttpClient.java:124) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)] at com.clickhouse.client.AbstractClient.sendAsync(AbstractClient.java:161) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)] at com.clickhouse.client.AbstractClient.lambda$execute$0(AbstractClient.java:273) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)] at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590) ~[?:1.8.0_102] ... 3 more Suppressed: java.io.IOException: Error writing request body to server at sun.net.www.protocol.http.HttpURLConnection$StreamingOutputStream.checkError(HttpURLConnection.java:3479) ~[?:1.8.0_102] at sun.net.www.protocol.http.HttpURLConnection$StreamingOutputStream.write(HttpURLConnection.java:3462) ~[?:1.8.0_102] at com.clickhouse.data.stream.Lz4OutputStream.flushBuffer(Lz4OutputStream.java:34) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)] at com.clickhouse.data.stream.Lz4OutputStream.flush(Lz4OutputStream.java:82) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)] at com.clickhouse.data.ClickHouseOutputStream.close(ClickHouseOutputStream.java:273) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)] at com.clickhouse.data.stream.Lz4OutputStream.close(Lz4OutputStream.java:92) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)] at com.clickhouse.client.http.ClickHouseHttpConnection.postData(ClickHouseHttpConnection.java:266) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)] at com.clickhouse.client.http.HttpUrlConnectionImpl.post(HttpUrlConnectionImpl.java:225) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)] at com.clickhouse.client.http.ClickHouseHttpClient.send(ClickHouseHttpClient.java:124) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)] at com.clickhouse.client.AbstractClient.sendAsync(AbstractClient.java:161) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)] at com.clickhouse.client.AbstractClient.lambda$execute$0(AbstractClient.java:273) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)] at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590) ~[?:1.8.0_102] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[?:1.8.0_102] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[?:1.8.0_102] at java.lang.Thread.run(Thread.java:745) ~[?:1.8.0_102] 18:49:17.814 [Executor task launch worker for task 0.4 in stage 7.0 (TID 887)] ERROR org.apache.spark.sql.execution.datasources.v2.DataWritingSparkTask - Aborting commit for partition 0 (task 887, attempt 4, stage 7.0) 18:49:17.814 [Executor task launch worker for task 0.4 in stage 7.0 (TID 887)] ERROR org.apache.spark.sql.execution.datasources.v2.DataWritingSparkTask - Aborted commit for partition 0 (task 887, attempt 4, stage 7.0) 18:49:17.815 [Executor task launch worker for task 0.4 in stage 7.0 (TID 887)] ERROR org.apache.spark.executor.Executor - Exception in task 0.4 in stage 7.0 (TID 887) xenon.clickhouse.exception.CHServerException: [HTTP]user@2.clickhouse.host:8123}/clustername [1002] Error writing request body to server, server ClickHouseNode [uri=http://2.clickhouse.host:8123/clustername, options={socket_timeout=600000,socket_keepalive=true,session_timeout=1200}]@-412378870 at xenon.clickhouse.client.NodeClient.syncInsert(NodeClient.scala:121) ~[clickhouse-spark-runtime-3.3_2.12-0.7.3.jar:?] at xenon.clickhouse.client.NodeClient.syncInsertOutputJSONEachRow(NodeClient.scala:86) ~[clickhouse-spark-runtime-3.3_2.12-0.7.3.jar:?] at xenon.clickhouse.write.ClickHouseWriter.$anonfun$doFlush$1(ClickHouseWriter.scala:232) ~[clickhouse-spark-runtime-3.3_2.12-0.7.3.jar:?] at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23) ~[scala-library-2.12.15.jar:?] at scala.util.Try$.apply(Try.scala:213) ~[scala-library-2.12.15.jar:?] at xenon.clickhouse.Utils$.retry(Utils.scala:99) ~[clickhouse-spark-runtime-3.3_2.12-0.7.3.jar:?] at xenon.clickhouse.write.ClickHouseWriter.doFlush(ClickHouseWriter.scala:230) ~[clickhouse-spark-runtime-3.3_2.12-0.7.3.jar:?] at xenon.clickhouse.write.ClickHouseWriter.flush(ClickHouseWriter.scala:220) ~[clickhouse-spark-runtime-3.3_2.12-0.7.3.jar:?] at xenon.clickhouse.write.ClickHouseWriter.write(ClickHouseWriter.scala:192) ~[clickhouse-spark-runtime-3.3_2.12-0.7.3.jar:?] at xenon.clickhouse.write.ClickHouseWriter.write(ClickHouseWriter.scala:39) ~[clickhouse-spark-runtime-3.3_2.12-0.7.3.jar:?] at org.apache.spark.sql.execution.datasources.v2.DataWritingSparkTask$.$anonfun$run$1(WriteToDataSourceV2Exec.scala:442) ~[spark-sql_2.12-3.3.0.jar:3.3.0] at org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1538) ~[spark-core_2.12-3.3.0.jar:3.3.0] at org.apache.spark.sql.execution.datasources.v2.DataWritingSparkTask$.run(WriteToDataSourceV2Exec.scala:480) ~[spark-sql_2.12-3.3.0.jar:3.3.0] at org.apache.spark.sql.execution.datasources.v2.V2TableWriteExec.$anonfun$writeWithV2$2(WriteToDataSourceV2Exec.scala:381) ~[spark-sql_2.12-3.3.0.jar:3.3.0] at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90) ~[spark-core_2.12-3.3.0.jar:3.3.0] at org.apache.spark.scheduler.Task.run(Task.scala:136) ~[spark-core_2.12-3.3.0.jar:3.3.0] at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:548) ~[spark-core_2.12-3.3.0.jar:3.3.0] at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1504) ~[spark-core_2.12-3.3.0.jar:3.3.0] at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:551) ~[spark-core_2.12-3.3.0.jar:3.3.0] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[?:1.8.0_102] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[?:1.8.0_102] at java.lang.Thread.run(Thread.java:745) ~[?:1.8.0_102] Caused by: com.clickhouse.client.ClickHouseException: Error writing request body to server, server ClickHouseNode [uri=http://2.clickhouse.host:8123/clustername, options={socket_timeout=600000,socket_keepalive=true,session_timeout=1200}]@-412378870 at com.clickhouse.client.ClickHouseException.of(ClickHouseException.java:168) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)] at com.clickhouse.client.AbstractClient.lambda$execute$0(AbstractClient.java:275) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)] at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590) ~[?:1.8.0_102] ... 3 more Caused by: java.io.IOException: Error writing request body to server at sun.net.www.protocol.http.HttpURLConnection$StreamingOutputStream.checkError(HttpURLConnection.java:3479) ~[?:1.8.0_102] at sun.net.www.protocol.http.HttpURLConnection$StreamingOutputStream.write(HttpURLConnection.java:3462) ~[?:1.8.0_102] at com.clickhouse.data.stream.Lz4OutputStream.flushBuffer(Lz4OutputStream.java:34) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)] at com.clickhouse.data.stream.AbstractByteArrayOutputStream.writeBuffer(AbstractByteArrayOutputStream.java:79) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)] at com.clickhouse.data.ClickHouseInputStream.pipe(ClickHouseInputStream.java:889) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)] at com.clickhouse.data.ClickHouseInputStream.pipe(ClickHouseInputStream.java:504) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)] at com.clickhouse.client.http.ClickHouseHttpConnection.postData(ClickHouseHttpConnection.java:318) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)] at com.clickhouse.client.http.HttpUrlConnectionImpl.post(HttpUrlConnectionImpl.java:225) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)] at com.clickhouse.client.http.ClickHouseHttpClient.send(ClickHouseHttpClient.java:124) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)] at com.clickhouse.client.AbstractClient.sendAsync(AbstractClient.java:161) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)] at com.clickhouse.client.AbstractClient.lambda$execute$0(AbstractClient.java:273) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)] at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590) ~[?:1.8.0_102] ... 3 more Suppressed: java.io.IOException: Error writing request body to server at sun.net.www.protocol.http.HttpURLConnection$StreamingOutputStream.checkError(HttpURLConnection.java:3479) ~[?:1.8.0_102] at sun.net.www.protocol.http.HttpURLConnection$StreamingOutputStream.write(HttpURLConnection.java:3462) ~[?:1.8.0_102] at com.clickhouse.data.stream.Lz4OutputStream.flushBuffer(Lz4OutputStream.java:34) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)] at com.clickhouse.data.stream.Lz4OutputStream.flush(Lz4OutputStream.java:82) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)] at com.clickhouse.data.ClickHouseOutputStream.close(ClickHouseOutputStream.java:273) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)] at com.clickhouse.data.stream.Lz4OutputStream.close(Lz4OutputStream.java:92) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)] at com.clickhouse.client.http.ClickHouseHttpConnection.postData(ClickHouseHttpConnection.java:266) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)] at com.clickhouse.client.http.HttpUrlConnectionImpl.post(HttpUrlConnectionImpl.java:225) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)] at com.clickhouse.client.http.ClickHouseHttpClient.send(ClickHouseHttpClient.java:124) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)] at com.clickhouse.client.AbstractClient.sendAsync(AbstractClient.java:161) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)] at com.clickhouse.client.AbstractClient.lambda$execute$0(AbstractClient.java:273) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)] at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590) ~[?:1.8.0_102] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[?:1.8.0_102] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[?:1.8.0_102] at java.lang.Thread.run(Thread.java:745) ~[?:1.8.0_102] 18:49:18.122 [Executor task launch worker for task 18.0 in stage 7.0 (TID 895)] ERROR org.apache.spark.util.Utils - Aborting task xenon.clickhouse.exception.CHServerException: [HTTP]user@1.clickhouse.host:8123}/clustername [1002] Error writing request body to server, server ClickHouseNode [uri=http://1.clickhouse.host:8123/clustername, options={socket_timeout=600000,socket_keepalive=true,session_timeout=1200}]@251648836 at xenon.clickhouse.client.NodeClient.syncInsert(NodeClient.scala:121) ~[clickhouse-spark-runtime-3.3_2.12-0.7.3.jar:?] at xenon.clickhouse.client.NodeClient.syncInsertOutputJSONEachRow(NodeClient.scala:86) ~[clickhouse-spark-runtime-3.3_2.12-0.7.3.jar:?] at xenon.clickhouse.write.ClickHouseWriter.$anonfun$doFlush$1(ClickHouseWriter.scala:232) ~[clickhouse-spark-runtime-3.3_2.12-0.7.3.jar:?] at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23) ~[scala-library-2.12.15.jar:?] at scala.util.Try$.apply(Try.scala:213) ~[scala-library-2.12.15.jar:?] at xenon.clickhouse.Utils$.retry(Utils.scala:99) ~[clickhouse-spark-runtime-3.3_2.12-0.7.3.jar:?] at xenon.clickhouse.write.ClickHouseWriter.doFlush(ClickHouseWriter.scala:230) ~[clickhouse-spark-runtime-3.3_2.12-0.7.3.jar:?] at xenon.clickhouse.write.ClickHouseWriter.flush(ClickHouseWriter.scala:218) ~[clickhouse-spark-runtime-3.3_2.12-0.7.3.jar:?] at xenon.clickhouse.write.ClickHouseWriter.commit(ClickHouseWriter.scala:263) ~[clickhouse-spark-runtime-3.3_2.12-0.7.3.jar:?] at org.apache.spark.sql.execution.datasources.v2.DataWritingSparkTask$.$anonfun$run$1(WriteToDataSourceV2Exec.scala:453) ~[spark-sql_2.12-3.3.0.jar:3.3.0] at org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1538) ~[spark-core_2.12-3.3.0.jar:3.3.0] at org.apache.spark.sql.execution.datasources.v2.DataWritingSparkTask$.run(WriteToDataSourceV2Exec.scala:480) ~[spark-sql_2.12-3.3.0.jar:3.3.0] at org.apache.spark.sql.execution.datasources.v2.V2TableWriteExec.$anonfun$writeWithV2$2(WriteToDataSourceV2Exec.scala:381) ~[spark-sql_2.12-3.3.0.jar:3.3.0] at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90) ~[spark-core_2.12-3.3.0.jar:3.3.0] at org.apache.spark.scheduler.Task.run(Task.scala:136) ~[spark-core_2.12-3.3.0.jar:3.3.0] at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:548) ~[spark-core_2.12-3.3.0.jar:3.3.0] at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1504) ~[spark-core_2.12-3.3.0.jar:3.3.0] at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:551) ~[spark-core_2.12-3.3.0.jar:3.3.0] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[?:1.8.0_102] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[?:1.8.0_102] at java.lang.Thread.run(Thread.java:745) ~[?:1.8.0_102] Caused by: com.clickhouse.client.ClickHouseException: Error writing request body to server, server ClickHouseNode [uri=http://1.clickhouse.host:8123/clustername, options={socket_timeout=600000,socket_keepalive=true,session_timeout=1200}]@251648836 at com.clickhouse.client.ClickHouseException.of(ClickHouseException.java:168) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)] at com.clickhouse.client.AbstractClient.lambda$execute$0(AbstractClient.java:275) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)] at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590) ~[?:1.8.0_102] ... 3 more Caused by: java.io.IOException: Error writing request body to server at sun.net.www.protocol.http.HttpURLConnection$StreamingOutputStream.checkError(HttpURLConnection.java:3479) ~[?:1.8.0_102] at sun.net.www.protocol.http.HttpURLConnection$StreamingOutputStream.write(HttpURLConnection.java:3462) ~[?:1.8.0_102] at com.clickhouse.data.stream.Lz4OutputStream.flushBuffer(Lz4OutputStream.java:51) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)] at com.clickhouse.data.stream.AbstractByteArrayOutputStream.writeBuffer(AbstractByteArrayOutputStream.java:80) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)] at com.clickhouse.data.ClickHouseInputStream.pipe(ClickHouseInputStream.java:889) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)] at com.clickhouse.data.ClickHouseInputStream.pipe(ClickHouseInputStream.java:504) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)] at com.clickhouse.client.http.ClickHouseHttpConnection.postData(ClickHouseHttpConnection.java:318) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)] at com.clickhouse.client.http.HttpUrlConnectionImpl.post(HttpUrlConnectionImpl.java:225) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)] at com.clickhouse.client.http.ClickHouseHttpClient.send(ClickHouseHttpClient.java:124) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)] at com.clickhouse.client.AbstractClient.sendAsync(AbstractClient.java:161) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)] at com.clickhouse.client.AbstractClient.lambda$execute$0(AbstractClient.java:273) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)] at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590) ~[?:1.8.0_102] ... 3 more 18:49:18.124 [Executor task launch worker for task 18.0 in stage 7.0 (TID 895)] ERROR org.apache.spark.sql.execution.datasources.v2.DataWritingSparkTask - Aborting commit for partition 18 (task 895, attempt 0, stage 7.0) 18:49:18.124 [Executor task launch worker for task 18.0 in stage 7.0 (TID 895)] ERROR org.apache.spark.sql.execution.datasources.v2.DataWritingSparkTask - Aborted commit for partition 18 (task 895, attempt 0, stage 7.0) 18:49:18.125 [Executor task launch worker for task 18.0 in stage 7.0 (TID 895)] ERROR org.apache.spark.executor.Executor - Exception in task 18.0 in stage 7.0 (TID 895) xenon.clickhouse.exception.CHServerException: [HTTP]user@1.clickhouse.host:8123}/clustername [1002] Error writing request body to server, server ClickHouseNode [uri=http://1.clickhouse.host:8123/clustername, options={socket_timeout=600000,socket_keepalive=true,session_timeout=1200}]@251648836 at xenon.clickhouse.client.NodeClient.syncInsert(NodeClient.scala:121) ~[clickhouse-spark-runtime-3.3_2.12-0.7.3.jar:?] at xenon.clickhouse.client.NodeClient.syncInsertOutputJSONEachRow(NodeClient.scala:86) ~[clickhouse-spark-runtime-3.3_2.12-0.7.3.jar:?] at xenon.clickhouse.write.ClickHouseWriter.$anonfun$doFlush$1(ClickHouseWriter.scala:232) ~[clickhouse-spark-runtime-3.3_2.12-0.7.3.jar:?] at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23) ~[scala-library-2.12.15.jar:?] at scala.util.Try$.apply(Try.scala:213) ~[scala-library-2.12.15.jar:?] at xenon.clickhouse.Utils$.retry(Utils.scala:99) ~[clickhouse-spark-runtime-3.3_2.12-0.7.3.jar:?] at xenon.clickhouse.write.ClickHouseWriter.doFlush(ClickHouseWriter.scala:230) ~[clickhouse-spark-runtime-3.3_2.12-0.7.3.jar:?] at xenon.clickhouse.write.ClickHouseWriter.flush(ClickHouseWriter.scala:218) ~[clickhouse-spark-runtime-3.3_2.12-0.7.3.jar:?] at xenon.clickhouse.write.ClickHouseWriter.commit(ClickHouseWriter.scala:263) ~[clickhouse-spark-runtime-3.3_2.12-0.7.3.jar:?] at org.apache.spark.sql.execution.datasources.v2.DataWritingSparkTask$.$anonfun$run$1(WriteToDataSourceV2Exec.scala:453) ~[spark-sql_2.12-3.3.0.jar:3.3.0] at org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1538) ~[spark-core_2.12-3.3.0.jar:3.3.0] at org.apache.spark.sql.execution.datasources.v2.DataWritingSparkTask$.run(WriteToDataSourceV2Exec.scala:480) ~[spark-sql_2.12-3.3.0.jar:3.3.0] at org.apache.spark.sql.execution.datasources.v2.V2TableWriteExec.$anonfun$writeWithV2$2(WriteToDataSourceV2Exec.scala:381) ~[spark-sql_2.12-3.3.0.jar:3.3.0] at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90) ~[spark-core_2.12-3.3.0.jar:3.3.0] at org.apache.spark.scheduler.Task.run(Task.scala:136) ~[spark-core_2.12-3.3.0.jar:3.3.0] at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:548) ~[spark-core_2.12-3.3.0.jar:3.3.0] at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1504) ~[spark-core_2.12-3.3.0.jar:3.3.0] at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:551) ~[spark-core_2.12-3.3.0.jar:3.3.0] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[?:1.8.0_102] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[?:1.8.0_102] at java.lang.Thread.run(Thread.java:745) ~[?:1.8.0_102] Caused by: com.clickhouse.client.ClickHouseException: Error writing request body to server, server ClickHouseNode [uri=http://1.clickhouse.host:8123/clustername, options={socket_timeout=600000,socket_keepalive=true,session_timeout=1200}]@251648836 at com.clickhouse.client.ClickHouseException.of(ClickHouseException.java:168) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)] at com.clickhouse.client.AbstractClient.lambda$execute$0(AbstractClient.java:275) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)] at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590) ~[?:1.8.0_102] ... 3 more Caused by: java.io.IOException: Error writing request body to server at sun.net.www.protocol.http.HttpURLConnection$StreamingOutputStream.checkError(HttpURLConnection.java:3479) ~[?:1.8.0_102] at sun.net.www.protocol.http.HttpURLConnection$StreamingOutputStream.write(HttpURLConnection.java:3462) ~[?:1.8.0_102] at com.clickhouse.data.stream.Lz4OutputStream.flushBuffer(Lz4OutputStream.java:51) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)] at com.clickhouse.data.stream.AbstractByteArrayOutputStream.writeBuffer(AbstractByteArrayOutputStream.java:80) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)] at com.clickhouse.data.ClickHouseInputStream.pipe(ClickHouseInputStream.java:889) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)] at com.clickhouse.data.ClickHouseInputStream.pipe(ClickHouseInputStream.java:504) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)] at com.clickhouse.client.http.ClickHouseHttpConnection.postData(ClickHouseHttpConnection.java:318) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)] at com.clickhouse.client.http.HttpUrlConnectionImpl.post(HttpUrlConnectionImpl.java:225) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)] at com.clickhouse.client.http.ClickHouseHttpClient.send(ClickHouseHttpClient.java:124) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)] at com.clickhouse.client.AbstractClient.sendAsync(AbstractClient.java:161) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)] at com.clickhouse.client.AbstractClient.lambda$execute$0(AbstractClient.java:273) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)] at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590) ~[?:1.8.0_102] ... 3 more ```

ShishkovSpace commented 6 days ago

I'm not sure if you have awared this article, which introduces the generic idea of writing optimization. the key idea is

So, to achieve better performance, we should re-organize the DataFrame to fit ClickHouse data layout before inserting.

I noticed you have disabled spark.clickhouse.write.repartitionByPartition, then how did you organize your data before inserting? could you please paste your Spark's DataFrame DAG and the writing stage info?

Yes, sure, i disabled spark.clickhouse.write.repartitionByPartition because data already organized by partiton key. I've made some tests in which try to investigate impact of connector parameters and dataframe reorganization. Here are options which i have played:

In all of these cases i have faced approximately same results on writing stage

df_connector_option_rapartitionByPartition_true

Maybe i missed something but i didn't face any impact of spark.clickhouse.write.repartitionByPartition in my scenario, but i suppose that my connector's configuration is not correct for these cases.