sink exception e.flink.connector.clickhouse.internal.ClickHouseBatchOutputFormat

zhanbing123 commented 2 months ago

"clickhouse-batch-output-format-thread-1" Id=93 RUNNABLE at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.socketRead(SocketInputStream.java:116) at java.net.SocketInputStream.read(SocketInputStream.java:171) at java.net.SocketInputStream.read(SocketInputStream.java:141) at org.apache.flink.shaded.clickhouse.org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137) at org.apache.flink.shaded.clickhouse.org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153) at org.apache.flink.shaded.clickhouse.org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:280) at org.apache.flink.shaded.clickhouse.org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.flink.shaded.clickhouse.org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.flink.shaded.clickhouse.org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259) at org.apache.flink.shaded.clickhouse.org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163) at org.apache.flink.shaded.clickhouse.org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:157) at org.apache.flink.shaded.clickhouse.org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273) at org.apache.flink.shaded.clickhouse.org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125) at org.apache.flink.shaded.clickhouse.org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.flink.shaded.clickhouse.org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.flink.shaded.clickhouse.org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.flink.shaded.clickhouse.org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.flink.shaded.clickhouse.org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.apache.flink.shaded.clickhouse.ru.yandex.clickhouse.ClickHouseStatementImpl.sendStream(ClickHouseStatementImpl.java:1059) at org.apache.flink.shaded.clickhouse.ru.yandex.clickhouse.ClickHouseStatementImpl.sendStream(ClickHouseStatementImpl.java:1026) at org.apache.flink.shaded.clickhouse.ru.yandex.clickhouse.ClickHouseStatementImpl.sendStream(ClickHouseStatementImpl.java:1019) at org.apache.flink.shaded.clickhouse.ru.yandex.clickhouse.ClickHousePreparedStatementImpl.executeBatch(ClickHousePreparedStatementImpl.java:381) at org.apache.flink.shaded.clickhouse.ru.yandex.clickhouse.ClickHousePreparedStatementImpl.executeBatch(ClickHousePreparedStatementImpl.java:364) at org.apache.flink.connector.clickhouse.internal.connection.ClickHouseStatementWrapper.executeBatch(ClickHouseStatementWrapper.java:24) at org.apache.flink.connector.clickhouse.internal.executor.ClickHouseExecutor.attemptExecuteBatch(ClickHouseExecutor.java:48) at org.apache.flink.connector.clickhouse.internal.executor.ClickHouseUpsertExecutor.executeBatch(ClickHouseUpsertExecutor.java:127) at org.apache.flink.connector.clickhouse.internal.AbstractClickHouseOutputFormat.checkBeforeFlush(AbstractClickHouseOutputFormat.java:88) at org.apache.flink.connector.clickhouse.internal.ClickHouseBatchOutputFormat.flush(ClickHouseBatchOutputFormat.java:93) at org.apache.flink.connector.clickhouse.internal.AbstractClickHouseOutputFormat.lambda$scheduledFlush$0(AbstractClickHouseOutputFormat.java:73)

locked org.apache.flink.connector.clickhouse.internal.ClickHouseBatchOutputFormat@7e457243 at org.apache.flink.connector.clickhouse.internal.AbstractClickHouseOutputFormat$$Lambda$947/902296269.run(Unknown Source) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)

Number of locked synchronizers = 1
java.util.concurrent.ThreadPoolExecutor$Worker@35acc8e0

zhanbing123 commented 2 months ago

After running for a while, the Flink job stopped the webUI shows high back-pressure, and the thread dump indicates that the issue is related to ClickHouse lock.Problem will solved if restart FlinkJob, However, it will reoccur again.

czy006 commented 2 months ago

@zhanbing123 https://github.com/itinycheng/flink-connector-clickhouse/pull/121 The new version will use a new client that you can check out. Please provide the version information about the environment occurrence

czy006 commented 2 weeks ago

fix in #141

itinycheng / flink-connector-clickhouse

sink exception e.flink.connector.clickhouse.internal.ClickHouseBatchOutputFormat #123