search

polardb / polardbx-cdc

polardbx-cdc is a core component of PolarDB-X which is responsible for global binary log generation, publication and subscription.
Apache License 2.0
73 stars 44 forks source link

show slave status报io exception #2

Closed welyss closed 2 years ago

welyss commented 2 years ago

cdc版本用的是lastest，由于镜像版本和galaxycdc版本对应关系没找到，所以不知道lastest是哪个版本. mysql> show slave status\G ERROR 3009 (HY000): [14a1336302402000][10.42.7.85:3306][rpl]UNAVAILABLE: io exception 集群运行状态正常: [root@localhost polardb-x]# kubectl get pods NAME READY STATUS RESTARTS AGE pxtest-29nk-cdc-half-6f6664c556-nz8b8 2/2 Running 0 26m pxtest-29nk-cn-zone-a-67f46c7c86-jv5z5 3/3 Running 0 10m pxtest-29nk-dn-0-cands1-0 3/3 Running 0 6d1h pxtest-29nk-dn-0-cands2-0 3/3 Running 0 6d1h pxtest-29nk-dn-0-log-0 3/3 Running 0 6d1h pxtest-29nk-gms-cand-zone-a-0 3/3 Running 0 21m pxtest-29nk-gms-cand-zone-b-0 3/3 Running 0 20m pxtest-29nk-gms-log-zone-c-0 3/3 Running 0 19m

CDC错误日志:

ERROR 9001 (HY000) at line 534: [14a10a6ddf801000][10.42.9.50:3306][grafana]ERR-CODE: [PXC-9001][ERR_GMS_GENERIC] Identifier name '_local_UQE_library_element_connection_element_id_kind_connection_id' is too long (max = 64). 2022-07-20 17:51:14.863 [qtp300104620-42] INFO c.a.p.binlog.daemon.rest.RestServer - 127.0.0.1 - - [20/Jul/2022:09:51:14 +0000] "GET //127.0.0.1:3007/cdc/metrics HTTP/1.1" 200 6 9 2022-07-20 17:51:15.505 [TimerTask-NodeReport] ERROR c.a.p.b.task.AbstractBinlogTimerTask - exec timer task fail NodeReport 1000 org.springframework.jdbc.UncategorizedSQLException: ### Error updating database. Cause: java.sql.SQLException: The consensus follower is not allowed to to do current operation. ### The error may exist in com/aliyun/polardbx/binlog/dao/NodeInfoMapper.java (best guess) ### The error may involve com.aliyun.polardbx.binlog.dao.NodeInfoMapper.insert-Inline ### The error occurred while setting parameters ### SQL: insert into binlog_node_info (gmt_created, gmt_modified, cluster_id, container_id, ip, daemon_port, available_ports, status, core, mem, gmt_heartbeat, role, cluster_type) values (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) ### Cause: java.sql.SQLException: The consensus follower is not allowed to to do current operation. ; uncategorized SQLException for SQL []; SQL state [HY000]; error code [3934]; The consensus follower is not allowed to to do current operation.; nested exception is java.sql.SQLException: The consensus follower is not allowed to to do current operation. at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:84) at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:81) at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:81) at org.mybatis.spring.MyBatisExceptionTranslator.translateExceptionIfPossible(MyBatisExceptionTranslator.java:88) at org.mybatis.spring.SqlSessionTemplate$SqlSessionInterceptor.invoke(SqlSessionTemplate.java:440) at com.sun.proxy.$Proxy17.insert(Unknown Source) at org.mybatis.spring.SqlSessionTemplate.insert(SqlSessionTemplate.java:271) at org.apache.ibatis.binding.MapperMethod.execute(MapperMethod.java:62) at org.apache.ibatis.binding.MapperProxy$PlainMethodInvoker.invoke(MapperProxy.java:152) at org.apache.ibatis.binding.MapperProxy.invoke(MapperProxy.java:85) at com.sun.proxy.$Proxy31.insert(Unknown Source) at org.mybatis.dynamic.sql.util.mybatis3.MyBatis3Utils.insert(MyBatis3Utils.java:93) at com.aliyun.polardbx.binlog.dao.NodeInfoMapper.insertSelective(NodeInfoMapper.java:212) at java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:627) at org.apache.ibatis.binding.MapperProxy$DefaultMethodInvoker.invoke(MapperProxy.java:166) at org.apache.ibatis.binding.MapperProxy.invoke(MapperProxy.java:85) at com.sun.proxy.$Proxy31.insertSelective(Unknown Source) at com.aliyun.polardbx.binlog.daemon.schedule.NodeReporter.exec(NodeReporter.java:90) at com.aliyun.polardbx.binlog.task.AbstractBinlogTimerTask$1.run(AbstractBinlogTimerTask.java:55) at java.util.TimerThread.mainLoop(Timer.java:555) at java.util.TimerThread.run(Timer.java:505) Caused by: java.sql.SQLException: The consensus follower is not allowed to to do current operation. at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:965) at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3933) at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3869) at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2524) at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2675) at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2465) at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1915) at com.mysql.jdbc.PreparedStatement.execute(PreparedStatement.java:1254) at sun.reflect.GeneratedMethodAccessor27.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.tomcat.jdbc.pool.StatementFacade$StatementProxy.invoke(StatementFacade.java:114) at com.sun.proxy.$Proxy30.execute(Unknown Source) at org.apache.ibatis.executor.statement.PreparedStatementHandler.update(PreparedStatementHandler.java:47) at org.apache.ibatis.executor.statement.RoutingStatementHandler.update(RoutingStatementHandler.java:74) at org.apache.ibatis.executor.ReuseExecutor.doUpdate(ReuseExecutor.java:52) at org.apache.ibatis.executor.BaseExecutor.update(BaseExecutor.java:117) at org.apache.ibatis.executor.CachingExecutor.update(CachingExecutor.java:76) at org.apache.ibatis.session.defaults.DefaultSqlSession.update(DefaultSqlSession.java:197) at org.apache.ibatis.session.defaults.DefaultSqlSession.insert(DefaultSqlSession.java:184) at sun.reflect.GeneratedMethodAccessor55.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.mybatis.spring.SqlSessionTemplate$SqlSessionInterceptor.invoke(SqlSessionTemplate.java:426) ... 16 common frames omitted 2022-07-20 17:51:16.501 [TimerTask-NodeReport] ERROR c.a.p.b.task.AbstractBinlogTimerTask - exec timer task fail NodeReport 1000 org.springframework.jdbc.UncategorizedSQLException: ### Error updating database. Cause: java.sql.SQLException: The consensus follower is not allowed to to do current operation. ### The error may exist in com/aliyun/polardbx/binlog/dao/NodeInfoMapper.java (best guess) ### The error may involve com.aliyun.polardbx.binlog.dao.NodeInfoMapper.insert-Inline ### The error occurred while setting parameters ### SQL: insert into binlog_node_info (gmt_created, gmt_modified, cluster_id, container_id, ip, daemon_port, available_ports, status, core, mem, gmt_heartbeat, role, cluster_type) values (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) ### Cause: java.sql.SQLException: The consensus follower is not allowed to to do current operation. ; uncategorized SQLException for SQL []; SQL state [HY000]; error code [3934]; The consensus follower is not allowed to to do current operation.; nested exception is java.sql.SQLException: The consensus follower is not allowed to to do current operation. at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:84) at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:81) at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:81) at org.mybatis.spring.MyBatisExceptionTranslator.translateExceptionIfPossible(MyBatisExceptionTranslator.java:88) at org.mybatis.spring.SqlSessionTemplate$SqlSessionInterceptor.invoke(SqlSessionTemplate.java:440) at com.sun.proxy.$Proxy17.insert(Unknown Source) at org.mybatis.spring.SqlSessionTemplate.insert(SqlSessionTemplate.java:271) at org.apache.ibatis.binding.MapperMethod.execute(MapperMethod.java:62) at org.apache.ibatis.binding.MapperProxy$PlainMethodInvoker.invoke(MapperProxy.java:152) at org.apache.ibatis.binding.MapperProxy.invoke(MapperProxy.java:85) at com.sun.proxy.$Proxy31.insert(Unknown Source) at org.mybatis.dynamic.sql.util.mybatis3.MyBatis3Utils.insert(MyBatis3Utils.java:93) at com.aliyun.polardbx.binlog.dao.NodeInfoMapper.insertSelective(NodeInfoMapper.java:212) at java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:627) at org.apache.ibatis.binding.MapperProxy$DefaultMethodInvoker.invoke(MapperProxy.java:166) at org.apache.ibatis.binding.MapperProxy.invoke(MapperProxy.java:85) at com.sun.proxy.$Proxy31.insertSelective(Unknown Source) at com.aliyun.polardbx.binlog.daemon.schedule.NodeReporter.exec(NodeReporter.java:90) at com.aliyun.polardbx.binlog.task.AbstractBinlogTimerTask$1.run(AbstractBinlogTimerTask.java:55) at java.util.TimerThread.mainLoop(Timer.java:555) at java.util.TimerThread.run(Timer.java:505) Caused by: java.sql.SQLException: The consensus follower is not allowed to to do current operation. at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:965) at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3933) at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3869) at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2524) at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2675) at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2465) at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1915) at com.mysql.jdbc.PreparedStatement.execute(PreparedStatement.java:1254) at sun.reflect.GeneratedMethodAccessor27.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.tomcat.jdbc.pool.StatementFacade$StatementProxy.invoke(StatementFacade.java:114) at com.sun.proxy.$Proxy30.execute(Unknown Source) at org.apache.ibatis.executor.statement.PreparedStatementHandler.update(PreparedStatementHandler.java:47) at org.apache.ibatis.executor.statement.RoutingStatementHandler.update(RoutingStatementHandler.java:74) at org.apache.ibatis.executor.ReuseExecutor.doUpdate(ReuseExecutor.java:52) at org.apache.ibatis.executor.BaseExecutor.update(BaseExecutor.java:117) at org.apache.ibatis.executor.CachingExecutor.update(CachingExecutor.java:76) at org.apache.ibatis.session.defaults.DefaultSqlSession.update(DefaultSqlSession.java:197) at org.apache.ibatis.session.defaults.DefaultSqlSession.insert(DefaultSqlSession.java:184) at sun.reflect.GeneratedMethodAccessor55.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.mybatis.spring.SqlSessionTemplate$SqlSessionInterceptor.invoke(SqlSessionTemplate.java:426) ... 16 common frames omitted

welyss commented 2 years ago

删除cdc，cn的pod后，过了段时间恢复了

Agilulf commented 2 years ago

报错的意思是连到gms 3节点的只读节点了，所以写sql无法执行，可能那个时间段发生了主备切换。 下次遇到执行 reload datasources 试试

welyss commented 2 years ago

@Agilulf 多谢，顺便问下，reload datasources的作用是cn节点的连接池刷新所有到dn节点的连接吗

Agilulf commented 2 years ago

@Agilulf 多谢，顺便问下，reload datasources的作用是cn节点的连接池刷新所有到dn节点的连接吗

是的