alibaba / canal

阿里巴巴 MySQL binlog 增量订阅&消费组件
Apache License 2.0
28.33k stars 7.58k forks source link

fastjson序列化时,内存溢出 #5214

Open duguwo opened 1 month ago

duguwo commented 1 month ago

数据库版本:5.7.33 canal版本:1.1.7 我的canal从1.1.4升级到1.1.7,几天运行后出现如下报错: 2024-07-20 10:17:02.923 [pool-4-thread-2] ERROR c.a.o.canal.connector.kafka.producer.CanalKafkaProducer - java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError at com.alibaba.otter.canal.common.utils.ExecutorTemplate.waitForResult(ExecutorTemplate.java:74) at com.alibaba.otter.canal.connector.kafka.producer.CanalKafkaProducer.send(CanalKafkaProducer.java:165) at com.alibaba.otter.canal.connector.core.spi.ProxyCanalMQProducer.send(ProxyCanalMQProducer.java:51) at com.alibaba.otter.canal.server.CanalMQStarter.worker(CanalMQStarter.java:181) at com.alibaba.otter.canal.server.CanalMQStarter.access$100(CanalMQStarter.java:24) at com.alibaba.otter.canal.server.CanalMQStarter$CanalMQRunnable.run(CanalMQStarter.java:223) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.alibaba.otter.canal.common.utils.ExecutorTemplate.waitForResult(ExecutorTemplate.java:72) ... 8 common frames omitted Caused by: java.lang.OutOfMemoryError: null at com.alibaba.fastjson2.JSONWriterUTF8.ensureCapacity(JSONWriterUTF8.java:1543) at com.alibaba.fastjson2.JSONWriterUTF8.writeString(JSONWriterUTF8.java:594) at com.alibaba.fastjson2.writer.OWG_5_14_FlatMessage.write(Unknown Source) at com.alibaba.fastjson2.JSON.toJSONBytes(JSON.java:2656) at com.alibaba.otter.canal.connector.kafka.producer.CanalKafkaProducer.send(CanalKafkaProducer.java:257) at com.alibaba.otter.canal.connector.kafka.producer.CanalKafkaProducer.lambda$send$0(CanalKafkaProducer.java:158) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted

我发现有其他伙伴出现同样问题https://github.com/alibaba/canal/issues/4865,但问题已经关闭,说是在主干已修复,是需要升级到1.1.8版本么?1.1.8正式版预计什么时候发布?谢谢!

twodog-tt commented 1 month ago

@duguwo 4865的抛错是ERROR c.a.o.c.c.rabbitmq.producer.CanalRabbitMQProducer - java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError,和你的CanalKafkaProducer不一致。

duguwo commented 1 month ago

@duguwo 4865的抛错是ERROR c.a.o.c.c.rabbitmq.producer.CanalRabbitMQProducer - java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError,和你的CanalKafkaProducer不一致。

我认为问题应该是一致的,只是使用的消息中间件不一样而已