search

datavane / tis

Support agile DataOps Based on Flink, DataX and Flink-CDC, Chunjun with Web-UI
https://tis.pub
Apache License 2.0
1.04k stars 221 forks source link

实时同步运行一段时间后挂掉 #363

Open bibichuan opened 1 month ago

bibichuan commented 1 month ago

我创建了两个实时同步的实例，从mysql80 到mysql80,刚创建的时候，可以实现实时同步，但是过了一天之后，就挂掉了 c9c0d0bad74f23fbdcbdeb0a328e6b95 这里是部分的日志文件 taskmanager_192.168.80.3_37823-2e2688_log.log

com.mysql.cj.jdbc.exceptions.CommunicationsException: The last packet successfully received from the server was 60,200,415 milliseconds ago. The last packet sent successfully to the server was 60,200,415 milliseconds ago. is longer than the server configured value of 'wait_timeout'. You should consider either expiring and/or testing connection validity before use in your application, increasing the server configured values for client timeouts, or using the Connector/J connection property 'autoReconnect=true' to avoid this problem.
    at com.mysql.cj.jdbc.exceptions.SQLError.createCommunicationsException(SQLError.java:174) ~[mysql-connector-j-8.0.31.jar:8.0.31]
    at com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:64) ~[mysql-connector-j-8.0.31.jar:8.0.31]
    at com.mysql.cj.jdbc.ConnectionImpl.isReadOnly(ConnectionImpl.java:1402) ~[mysql-connector-j-8.0.31.jar:8.0.31]
    at com.mysql.cj.jdbc.ConnectionImpl.isReadOnly(ConnectionImpl.java:1387) ~[mysql-connector-j-8.0.31.jar:8.0.31]
    at com.mysql.cj.jdbc.ClientPreparedStatement.executeBatchInternal(ClientPreparedStatement.java:386) ~[mysql-connector-j-8.0.31.jar:8.0.31]
    at com.mysql.cj.jdbc.StatementImpl.executeBatch(StatementImpl.java:795) ~[mysql-connector-j-8.0.31.jar:8.0.31]
    at org.apache.flink.connector.jdbc.statement.FieldNamedPreparedStatementImpl.executeBatch(FieldNamedPreparedStatementImpl.java:65) ~[flink-connector-jdbc-3.1.1-1.17.jar:3.1.1-1.17]
    at com.dtstack.chunjun.connector.jdbc.sink.PreparedStmtProxy.executeBatch(PreparedStmtProxy.java:305) ~[chunjun-connector-jdbc-base-1.12.5.jar:?]
    at com.dtstack.chunjun.connector.jdbc.sink.JdbcOutputFormat.writeMultipleRecordsInternal(JdbcOutputFormat.java:257) ~[chunjun-connector-jdbc-base-1.12.5.jar:?]
    at com.dtstack.chunjun.sink.format.BaseRichOutputFormat.writeRecordInternal(BaseRichOutputFormat.java:500) ~[chunjun-core-1.12.5.jar:?]
    at com.dtstack.chunjun.sink.format.BaseRichOutputFormat.lambda$initTimingSubmitTask$1(BaseRichOutputFormat.java:460) ~[chunjun-core-1.12.5.jar:?]
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_412]
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_412]
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_412]
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_412]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_412]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_412]
    at java.lang.Thread.run(Thread.java:750) [?:1.8.0_412]
Caused by: com.mysql.cj.exceptions.CJCommunicationsException: The last packet successfully received from the server was 60,200,415 milliseconds ago. The last packet sent successfully to the server was 60,200,415 milliseconds ago. is longer than the server configured value of 'wait_timeout'. You should consider either expiring and/or testing connection validity before use in your application, increasing the server configured values for client timeouts, or using the Connector/J connection property 'autoReconnect=true' to avoid this problem.
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:1.8.0_412]
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:1.8.0_412]
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:1.8.0_412]
    at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_412]
    at com.mysql.cj.exceptions.ExceptionFactory.createException(ExceptionFactory.java:61) ~[mysql-connector-j-8.0.31.jar:8.0.31]
    at com.mysql.cj.exceptions.ExceptionFactory.createException(ExceptionFactory.java:105) ~[mysql-connector-j-8.0.31.jar:8.0.31]
    at com.mysql.cj.exceptions.ExceptionFactory.createException(ExceptionFactory.java:151) ~[mysql-connector-j-8.0.31.jar:8.0.31]
    at com.mysql.cj.exceptions.ExceptionFactory.createCommunicationsException(ExceptionFactory.java:167) ~[mysql-connector-j-8.0.31.jar:8.0.31]
    at com.mysql.cj.protocol.a.NativeProtocol.send(NativeProtocol.java:628) ~[mysql-connector-j-8.0.31.jar:8.0.31]
    at com.mysql.cj.protocol.a.NativeProtocol.sendCommand(NativeProtocol.java:683) ~[mysql-connector-j-8.0.31.jar:8.0.31]
    at com.mysql.cj.protocol.a.NativeProtocol.sendCommand(NativeProtocol.java:155) ~[mysql-connector-j-8.0.31.jar:8.0.31]
    at com.mysql.cj.NativeSession.queryServerVariable(NativeSession.java:597) ~[mysql-connector-j-8.0.31.jar:8.0.31]
    at com.mysql.cj.jdbc.ConnectionImpl.isReadOnly(ConnectionImpl.java:1394) ~[mysql-connector-j-8.0.31.jar:8.0.31]
    ... 15 more
Caused by: java.net.SocketException: Broken pipe (Write failed)
    at java.net.SocketOutputStream.socketWrite0(Native Method) ~[?:1.8.0_412]
    at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:111) ~[?:1.8.0_412]
    at java.net.SocketOutputStream.write(SocketOutputStream.java:155) ~[?:1.8.0_412]
    at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82) ~[?:1.8.0_412]
    at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140) ~[?:1.8.0_412]
    at com.mysql.cj.protocol.a.CompressedPacketSender.send(CompressedPacketSender.java:188) ~[mysql-connector-j-8.0.31.jar:8.0.31]
    at com.mysql.cj.protocol.a.TimeTrackingPacketSender.send(TimeTrackingPacketSender.java:50) ~[mysql-connector-j-8.0.31.jar:8.0.31]
    at com.mysql.cj.protocol.a.NativeProtocol.send(NativeProtocol.java:619) ~[mysql-connector-j-8.0.31.jar:8.0.31]
    at com.mysql.cj.protocol.a.NativeProtocol.sendCommand(NativeProtocol.java:683) ~[mysql-connector-j-8.0.31.jar:8.0.31]
    at com.mysql.cj.protocol.a.NativeProtocol.sendCommand(NativeProtocol.java:155) ~[mysql-connector-j-8.0.31.jar:8.0.31]
    at com.mysql.cj.NativeSession.queryServerVariable(NativeSession.java:597) ~[mysql-connector-j-8.0.31.jar:8.0.31]
    at com.mysql.cj.jdbc.ConnectionImpl.isReadOnly(ConnectionImpl.java:1394) ~[mysql-connector-j-8.0.31.jar:8.0.31]
    ... 15 more
2024-09-24 10:45:42,208 WARN  com.dtstack.chunjun.dirty.log.LogDirtyDataCollector
baisui1981 commented 1 month ago

看上面异常是在sink端抛出的异常，这样好办。只要在数据源配置一项中，设置autoReconnect=true参数应该就可以了 截屏2024-09-24 15 52 46

baisui1981 commented 1 month ago

用户通过添加以上reconnect参数经过一段时间测试发现没有效果，依然会断连接。 继续查看文档还需要额外添加一些参数，文档：https://www.cnblogs.com/yhtboke/p/5632757.html

autoReconnect=true&timeBetweenEvictionRunsMillis=26400&testWhileIdle=true&validationQuery=SELECT%201

bibichuan commented 1 month ago

尝试找到数据库的配置文件 /usr/local/tis/docker-compose/data/cfg_repo/tis_plugin_config/db 目录，里面有各个数据库的配置文件，修改其中的xml,修改其中的 extraParams 参数，然后重启 tis，结果也是报错了：XmlPullParserException: entity reference name can not contain character =' (position: START_TAG seen

autoReconnect=true&timeBetweenEvictionRunsMillis=26400&testWhileIdle=true&validationQuery=SELECT%201