search

itinycheng / flink-connector-clickhouse

Flink SQL connector for ClickHouse. Support ClickHouseCatalog and read/write primary data, maps, arrays to clickhouse.
Apache License 2.0
349 stars 149 forks source link

ClickHouse executeBatch error #26

Closed kongyuan1990 closed 2 years ago

kongyuan1990 commented 2 years ago

when I use flink-connector-clickhouse. image

The following exceptions occur occasionally

2022-05-09 12:26:29,777 ERROR org.apache.flink.connector.clickhouse.internal.AbstractClickHouseOutputFormat [] - ClickHouse executeBatch error, retry times = 0 ru.yandex.clickhouse.except.ClickHouseUnknownException: ClickHouse exception, code: 1002, host: clickhouse-prod.xxx.com, port: 8123; null at ru.yandex.clickhouse.except.ClickHouseExceptionSpecifier.getException(ClickHouseExceptionSpecifier.java:92) ~[flink-connector-clickhouse-1.12.0.jar:?] at ru.yandex.clickhouse.except.ClickHouseExceptionSpecifier.specify(ClickHouseExceptionSpecifier.java:42) ~[flink-connector-clickhouse-1.12.0.jar:?] at ru.yandex.clickhouse.except.ClickHouseExceptionSpecifier.specify(ClickHouseExceptionSpecifier.java:25) ~[flink-connector-clickhouse-1.12.0.jar:?] at ru.yandex.clickhouse.ClickHouseStatementImpl.sendStream(ClickHouseStatementImpl.java:1071) ~[flink-connector-clickhouse-1.12.0.jar:?] at ru.yandex.clickhouse.ClickHouseStatementImpl.sendStream(ClickHouseStatementImpl.java:1026) ~[flink-connector-clickhouse-1.12.0.jar:?] at ru.yandex.clickhouse.ClickHouseStatementImpl.sendStream(ClickHouseStatementImpl.java:1019) ~[flink-connector-clickhouse-1.12.0.jar:?] at ru.yandex.clickhouse.ClickHousePreparedStatementImpl.executeBatch(ClickHousePreparedStatementImpl.java:381) ~[flink-connector-clickhouse-1.12.0.jar:?] at ru.yandex.clickhouse.ClickHousePreparedStatementImpl.executeBatch(ClickHousePreparedStatementImpl.java:364) ~[flink-connector-clickhouse-1.12.0.jar:?] at org.apache.flink.connector.clickhouse.internal.executor.ClickHouseBatchExecutor.executeBatch(ClickHouseBatchExecutor.java:72) ~[flink-connector-clickhouse-1.12.0.jar:?] at org.apache.flink.connector.clickhouse.internal.AbstractClickHouseOutputFormat.attemptFlush(AbstractClickHouseOutputFormat.java:84) ~[flink-connector-clickhouse-1.12.0.jar:?] at org.apache.flink.connector.clickhouse.internal.ClickHouseBatchOutputFormat.flush(ClickHouseBatchOutputFormat.java:93) ~[flink-connector-clickhouse-1.12.0.jar:?] at org.apache.flink.connector.clickhouse.internal.AbstractClickHouseOutputFormat.lambda$scheduledFlush$0(AbstractClickHouseOutputFormat.java:66) ~[flink-connector-clickhouse-1.12.0.jar:?] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_322] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_322] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_322] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_322] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_322] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_322] at java.lang.Thread.run(Thread.java:750) [?:1.8.0_322] Caused by: java.util.ConcurrentModificationException at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:911) ~[?:1.8.0_322] at java.util.ArrayList$Itr.next(ArrayList.java:861) ~[?:1.8.0_322] at ru.yandex.clickhouse.ClickHousePreparedStatementImpl$BatchHttpEntity.writeTo(ClickHousePreparedStatementImpl.java:419) ~[flink-connector-clickhouse-1.12.0.jar:?] at ru.yandex.clickhouse.ClickHouseStatementImpl$WrappedHttpEntity.writeTo(ClickHouseStatementImpl.java:98) ~[flink-connector-clickhouse-1.12.0.jar:?] at org.apache.http.impl.DefaultBHttpClientConnection.sendRequestEntity(DefaultBHttpClientConnection.java:156) ~[flink-connector-clickhouse-1.12.0.jar:?] at org.apache.http.impl.conn.CPoolProxy.sendRequestEntity(CPoolProxy.java:152) ~[flink-connector-clickhouse-1.12.0.jar:?] at org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:238) ~[flink-connector-clickhouse-1.12.0.jar:?] at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123) ~[flink-connector-clickhouse-1.12.0.jar:?] at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) ~[flink-connector-clickhouse-1.12.0.jar:?] at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) ~[flink-connector-clickhouse-1.12.0.jar:?] at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) ~[flink-connector-clickhouse-1.12.0.jar:?] at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[flink-connector-clickhouse-1.12.0.jar:?] at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) ~[flink-connector-clickhouse-1.12.0.jar:?] at ru.yandex.clickhouse.ClickHouseStatementImpl.sendStream(ClickHouseStatementImpl.java:1059) ~[flink-connector-clickhouse-1.12.0.jar:?] ... 15 more

kongyuan1990 commented 2 years ago

use tag release-1.12.0

kongyuan1990 commented 2 years ago

and, sink to clickhouse, lost some data But when I use datastream jdbc connector , No data loss occurred

itinycheng commented 2 years ago

This info is not enough for me to find the problem. If it happens accidentally, it may be caused by dirty data (I'm not sure).

This exception occurs during sending data to the server-side in clickhouse-jdbc.jar, you'd better do some breakpoint tests, where the exception occurred: image

kongyuan1990 commented 2 years ago

I tried three test cases.

case 1: use datastream jdbc connector;

5138574713_59063644815_09C7E806-84D3-437C-B58C-8EF244FD47F1

successed sink to clickhouse 9408 datas

case 2: use tag release-1.12.0

successed sink to clickhouse 8578 datas， lost 830 datas , no exception occurred

I still can't find the reason for the loss of data.

case 3: use this source code (https://github.com/liekkassmile/flink-connector-clickhouse)

successed sink to clickhouse 9408 datas

itinycheng commented 2 years ago

OK, can you email or WX me to talk some details about this?

kongyuan1990 commented 2 years ago

WX：snake_kong

itinycheng commented 2 years ago

Tag: release-1.12.0 has a thread-safe bug that causes this problem, bugfix commit: https://github.com/itinycheng/flink-connector-clickhouse/commit/bb4f078ef6f664746e539f9986c77f7cacdd4574, https://github.com/itinycheng/flink-connector-clickhouse/commit/345151584f743e8b77b448220bf4e0fbf89e7758 Use branch release-1.12 to avoid this issue and I have deleted the tag to prevent more people encounter this.