myql CDC encounters error Over max: 4289745

[huiyida]

Comment

版本：V2.14 报错信息如下： java.lang.RuntimeException: Unable to initialize and start connector's task class 'io.debezium.connector.mysql.MySqlConnectorTask' with config: {snapshot.mode=schema_only_recovery, database.user=, database.password=****, offset.storage=io.tapdata.connector.mysql.PdkPersistenceOffsetBackingStore, database.history.store.only.captured.tables.ddl=true, database.server.name=4345c6a6-054e-4cbe-aab7-9bd36611c51a, time.precision.mode=adaptive_time_microseconds, snapshot.locking.mode=none, pdk.offset.string={"name":"4345c6a6-054e-4cbe-aab7-9bd36611c51a","offset":{"{\"server\":\"4345c6a6-054e-4cbe-aab7-9bd36611c51a\"}":"{\"file\":\"mysql-bin.000491\",\"pos\":4246079,\"gtids\":\"ed7cc852-a5cb-11ed-a84e-b8599f496ab4:4289658-4289745\",\"server_id\":3274629774}"}}, max.queue.size=8000, connector.class=io.debezium.connector.mysql.MySqlConnector, database.port=3306, threadName=Debezium-Mysql-Connector-4345c6a6-054e-4cbe-aab7-9bd36611c51a, database.history.skip.unparseable.ddl=true, database.history.store.only.monitored.tables.ddl=true, database.hostname=sss123123.mysql.rds.aliyuncs.com, table.include.list=platform.city, database.include.list=platform, database.server.id=240436258, database.history=io.tapdata.connector.mysql.StateMapHistoryBackingStore, max.batch.size=1000, name=4345c6a6-054e-4cbe-aab7-9bd36611c51a} Over max: 4289745 at io.tapdata.connector.mysql.MysqlReader.lambda$readBinlog$6(MysqlReader.java:315) at io.debezium.embedded.EmbeddedEngine.run(EmbeddedEngine.java:891) at io.tapdata.connector.mysql.MysqlReader.readBinlog(MysqlReader.java:326) at io.tapdata.connector.mysql.MysqlConnector.streamRead(MysqlConnector.java:418) at io.tapdata.flow.engine.V2.node.hazelcast.data.pdk.HazelcastSourcePdkDataNode.lambda$null$11(HazelcastSourcePdkDataNode.java:501) at io.tapdata.pdk.core.monitor.PDKInvocationMonitor.invokePDKMethodPrivate(PDKInvocationMonitor.java:161) at io.tapdata.pdk.core.monitor.PDKInvocationMonitor.lambda$null$10(PDKInvocationMonitor.java:148) at io.tapdata.pdk.core.api.Node.applyClassLoaderContext(Node.java:27) at io.tapdata.pdk.core.monitor.PDKInvocationMonitor.lambda$invokePDKMethod$11(PDKInvocationMonitor.java:148) at io.tapdata.pdk.core.utils.RetryUtils.autoRetry(RetryUtils.java:62) at io.tapdata.pdk.core.monitor.PDKInvocationMonitor.invokePDKMethod(PDKInvocationMonitor.java:148) at io.tapdata.pdk.core.monitor.PDKInvocationMonitor.invoke(PDKInvocationMonitor.java:92) at io.tapdata.flow.engine.V2.node.hazelcast.data.pdk.HazelcastSourcePdkDataNode.lambda$doNormalCDC$12(HazelcastSourcePdkDataNode.java:424) at io.tapdata.aspect.utils.AspectUtils.executeDataFuncAspect(AspectUtils.java:67) at io.tapdata.flow.engine.V2.node.hazelcast.HazelcastBaseNode.executeDataFuncAspect(HazelcastBaseNode.java:197) at io.tapdata.flow.engine.V2.node.hazelcast.data.pdk.HazelcastSourcePdkDataNode.doNormalCDC(HazelcastSourcePdkDataNode.java:416) at io.tapdata.flow.engine.V2.node.hazelcast.data.pdk.HazelcastSourcePdkDataNode.doCdc(HazelcastSourcePdkDataNode.java:373) at io.tapdata.flow.engine.V2.node.hazelcast.data.pdk.HazelcastSourcePdkDataNode.startSourceRunner(HazelcastSourcePdkDataNode.java:148) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:266) at java.util.concurrent.FutureTask.run(FutureTask.java) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.RuntimeException: Over max: 4289745 at io.debezium.connector.mysql.utils.MergeGTIDUtils$Intervals.add(MergeGTIDUtils.java:93) at io.debezium.connector.mysql.utils.MergeGTIDUtils.add(MergeGTIDUtils.java:27) at io.debezium.connector.mysql.utils.MergeGTIDUtils.merge(MergeGTIDUtils.java:62) at io.debezium.connector.mysql.MySqlConnectorTask.start(MySqlConnectorTask.java:100) at io.debezium.connector.common.BaseSourceTask.start(BaseSourceTask.java:130) at io.debezium.embedded.EmbeddedEngine.run(EmbeddedEngine.java:759) ... 22 more

[tjworks]

This may have been fixed, please verify @xbsura