canal每天都有一次java.lang.OutOfMemoryError这个错误，并且运行一段时间后就kafka就收不到数据了，canal进程还在

[naturelyj]

阿里云RDS mysql5.7
canal抽取数据后写入kafka 下面问题是最近2天的错误日志 2024-07-02 07:33:05.158 [destination = zn , address = rm-.mysql.rds.aliyuncs.com/172.17.120.237:3306 , EventParser] ERROR c.a.o.c.p.inbound.mysql.rds.RdsBinlogEventParserProxy - dump address rm-2ze5h6891j393464g.mysql.rds.aliyuncs.com/172.17.120.237:3306 has an error, retrying. caused by com.alibaba.otter.canal.parse.exception.CanalParseException: com.googlecode.aviator.exception.ExpressionRuntimeException: Execute expression error Caused by: com.googlecode.aviator.exception.ExpressionRuntimeException: Execute expression error at com.googlecode.aviator.ClassExpression.execute(ClassExpression.java:59) at com.alibaba.otter.canal.filter.aviater.AviaterRegexFilter.filter(AviaterRegexFilter.java:73) at com.alibaba.otter.canal.parse.inbound.mysql.dbsync.LogEventConvert.parseRowsEventForTableMeta(LogEventConvert.java:466) at com.alibaba.otter.canal.parse.inbound.mysql.MysqlMultiStageCoprocessor$SimpleParserStage.processEvent(MysqlMultiStageCoprocessor.java:293) at com.alibaba.otter.canal.parse.inbound.mysql.MysqlMultiStageCoprocessor$SimpleParserStage.onEvent(MysqlMultiStageCoprocessor.java:276) at com.alibaba.otter.canal.parse.inbound.mysql.MysqlMultiStageCoprocessor$SimpleParserStage.onEvent(MysqlMultiStageCoprocessor.java:239) at com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:168) at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: com.google.common.util.concurrent.ExecutionError: java.lang.OutOfMemoryError: GC overhead limit exceeded at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2216) at com.google.common.cache.LocalCache.get(LocalCache.java:4147) at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:4151) at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:5140) at com.google.common.collect.MigrateMap$MigrateConcurrentMap.get(MigrateMap.java:68) at com.alibaba.otter.canal.filter.PatternUtils.getPattern(PatternUtils.java:29) at com.alibaba.otter.canal.filter.aviater.RegexFunction.call(RegexFunction.java:24) at Script_1719703765709_0.execute0(Unknown Source) at com.googlecode.aviator.ClassExpression.execute(ClassExpression.java:53) ... 12 common frames omitted Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded 2024-07-02 07:33:05.183 [destination = zn , address = rm-.mysql.rds.aliyuncs.com/172.17.120.237:3306 , EventParser] ERROR com.alibaba.otter.canal.common.alarm.LogAlarmHandler - destination:zn[com.alibaba.otter.canal.parse.exception.CanalParseException: com.googlecode.aviator.exception.ExpressionRuntimeException: Execute expression error Caused by: com.googlecode.aviator.exception.ExpressionRuntimeException: Execute expression error at com.googlecode.aviator.ClassExpression.execute(ClassExpression.java:59) at com.alibaba.otter.canal.filter.aviater.AviaterRegexFilter.filter(AviaterRegexFilter.java:73) at com.alibaba.otter.canal.parse.inbound.mysql.dbsync.LogEventConvert.parseRowsEventForTableMeta(LogEventConvert.java:466) at com.alibaba.otter.canal.parse.inbound.mysql.MysqlMultiStageCoprocessor$SimpleParserStage.processEvent(MysqlMultiStageCoprocessor.java:293) at com.alibaba.otter.canal.parse.inbound.mysql.MysqlMultiStageCoprocessor$SimpleParserStage.onEvent(MysqlMultiStageCoprocessor.java:276) at com.alibaba.otter.canal.parse.inbound.mysql.MysqlMultiStageCoprocessor$SimpleParserStage.onEvent(MysqlMultiStageCoprocessor.java:239) at com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:168) at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: com.google.common.util.concurrent.ExecutionError: java.lang.OutOfMemoryError: GC overhead limit exceeded at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2216) at com.google.common.cache.LocalCache.get(LocalCache.java:4147) at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:4151) at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:5140) at com.google.common.collect.MigrateMap$MigrateConcurrentMap.get(MigrateMap.java:68) at com.alibaba.otter.canal.filter.PatternUtils.getPattern(PatternUtils.java:29) at com.alibaba.otter.canal.filter.aviater.RegexFunction.call(RegexFunction.java:24) at Script_1719703765709_0.execute0(Unknown Source) at com.googlecode.aviator.ClassExpression.execute(ClassExpression.java:53) ... 12 more Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded ] 2024-07-02 07:33:09.622 [[scheduler-table-meta-snapshot]] ERROR c.a.o.canal.parse.inbound.mysql.tsdb.DatabaseTableMeta - scheudle applySnapshotToDB faield java.lang.OutOfMemoryError: Java heap space

2024-07-01 07:31:14.951 [[scheduler-table-meta-snapshot]] ERROR c.a.o.canal.parse.inbound.mysql.tsdb.DatabaseTableMeta - scheudle applySnapshotToDB faield java.lang.OutOfMemoryError: Java heap space at java.util.Arrays.copyOf(Arrays.java:3332) at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:124) at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:448) at java.lang.StringBuilder.append(StringBuilder.java:136) at ch.qos.logback.core.pattern.FormattingConverter.write(FormattingConverter.java:39) at ch.qos.logback.core.pattern.PatternLayoutBase.writeLoopOnConverters(PatternLayoutBase.java:115) at ch.qos.logback.classic.PatternLayout.doLayout(PatternLayout.java:165) at ch.qos.logback.classic.PatternLayout.doLayout(PatternLayout.java:39) at ch.qos.logback.core.encoder.LayoutWrappingEncoder.encode(LayoutWrappingEncoder.java:116) at ch.qos.logback.core.OutputStreamAppender.subAppend(OutputStreamAppender.java:230) at ch.qos.logback.core.rolling.RollingFileAppender.subAppend(RollingFileAppender.java:235) at ch.qos.logback.core.OutputStreamAppender.append(OutputStreamAppender.java:102) at ch.qos.logback.core.UnsynchronizedAppenderBase.doAppend(UnsynchronizedAppenderBase.java:84) at ch.qos.logback.core.sift.SiftingAppenderBase.append(SiftingAppenderBase.java:110) at ch.qos.logback.core.AppenderBase.doAppend(AppenderBase.java:82) at ch.qos.logback.core.spi.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:51) at ch.qos.logback.classic.Logger.appendLoopOnAppenders(Logger.java:270) at ch.qos.logback.classic.Logger.callAppenders(Logger.java:257) at ch.qos.logback.classic.Logger.buildLoggingEventAndAppend(Logger.java:421) at ch.qos.logback.classic.Logger.filterAndLog_0_Or3Plus(Logger.java:383) at ch.qos.logback.classic.Logger.warn(Logger.java:692) at com.alibaba.otter.canal.parse.inbound.mysql.tsdb.MemoryTableMeta.apply(MemoryTableMeta.java:88) at com.alibaba.otter.canal.parse.inbound.mysql.tsdb.DatabaseTableMeta.applySnapshotToDB(DatabaseTableMeta.java:310) at com.alibaba.otter.canal.parse.inbound.mysql.tsdb.DatabaseTableMeta.lambda$init$1(DatabaseTableMeta.java:94) at com.alibaba.otter.canal.parse.inbound.mysql.tsdb.DatabaseTableMeta$$Lambda$239/991563000.run(Unknown Source) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)

[agapple]

java.lang.OutOfMemoryError: Java heap space

先调大一些jvm内存试试，内存不足

[naturelyj]

java.lang.OutOfMemoryError: Java heap space

先调大一些jvm内存试试，内存不足

已调大，在监测中

[naturelyj]

java.lang.OutOfMemoryError: Java heap space

先调大一些jvm内存试试，内存不足

调大后，运行基本稳定，但是instance的log中每天差不多时间点会出现下列错误，会是什么原因呢？ [[scheduler-table-meta-snapshot]] ERROR c.a.o.canal.parse.inbound.mysql.tsdb.DatabaseTableMeta - scheudle applySnapshotToDB faield java.lang.OutOfMemoryError: null at com.alibaba.fastjson2.JSONWriterUTF16.ensureCapacity(JSONWriterUTF16.java:1250) at com.alibaba.fastjson2.JSONWriterUTF16.writeStringEscape(JSONWriterUTF16.java:366) at com.alibaba.fastjson2.JSONWriterUTF16JDK8UF.writeString(JSONWriterUTF16JDK8UF.java:165) at com.alibaba.fastjson2.writer.ObjectWriterImplMap.write(ObjectWriterImplMap.java:481) at com.alibaba.fastjson2.JSON.toJSONString(JSON.java:2373) at com.alibaba.otter.canal.parse.inbound.mysql.tsdb.DatabaseTableMeta.applySnapshotToDB(DatabaseTableMeta.java:337) at com.alibaba.otter.canal.parse.inbound.mysql.tsdb.DatabaseTableMeta.lambda$init$1(DatabaseTableMeta.java:94) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)