search

alibaba / canal

阿里巴巴 MySQL binlog 增量订阅&消费组件
Apache License 2.0
28.59k stars 7.63k forks source link

运行几个小时后，程序假死，server和adapter没有任何日志 #2986

Open aiguo123 opened 4 years ago

aiguo123 commented 4 years ago

Question

这个是线程栈 destination = lietou , address = /127.0.0.1:3306 , EventParser" Id=27 cpuUsage=66% TIMED_WAITING at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:338) at com.alibaba.otter.canal.parse.inbound.mysql.MysqlMultiStageCoprocessor.applyWait(MysqlMultiStageCoprocessor.java:241) at com.alibaba.otter.canal.parse.inbound.mysql.MysqlMultiStageCoprocessor.publish(MysqlMultiStageCoprocessor.java:223) at com.alibaba.otter.canal.parse.inbound.mysql.MysqlMultiStageCoprocessor.publish(MysqlMultiStageCoprocessor.java:175) at com.alibaba.otter.canal.parse.inbound.mysql.MysqlConnection.dump(MysqlConnection.java:239) at com.alibaba.otter.canal.parse.inbound.AbstractEventParser$3.run(AbstractEventParser.java:265) at java.lang.Thread.run(Thread.java:748)

"MultiStageCoprocessor-other-lietou-1" Id=31 cpuUsage=14% TIMED_WAITING at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:338) at com.alibaba.otter.canal.sink.entry.EntryEventSink.applyWait(EntryEventSink.java:197) at com.alibaba.otter.canal.sink.entry.EntryEventSink.doSink(EntryEventSink.java:175) at com.alibaba.otter.canal.sink.entry.EntryEventSink.sinkData(EntryEventSink.java:120) at com.alibaba.otter.canal.sink.entry.EntryEventSink.sink(EntryEventSink.java:86) at com.alibaba.otter.canal.sink.entry.EntryEventSink.sink(EntryEventSink.java:32) at com.alibaba.otter.canal.parse.inbound.AbstractEventParser.consumeTheEventAndProfilingIfNecessary(AbstractEventParser.java:398) at com.alibaba.otter.canal.parse.inbound.AbstractEventParser$2.flush(AbstractEventParser.java:140) at com.alibaba.otter.canal.parse.inbound.EventTransactionBuffer.flush(EventTransactionBuffer.java:123) at com.alibaba.otter.canal.parse.inbound.EventTransactionBuffer.add(EventTransactionBuffer.java:73) at com.alibaba.otter.canal.parse.inbound.mysql.MysqlMultiStageCoprocessor$SinkStoreStage.onEvent(MysqlMultiStageCoprocessor.java:357) at com.alibaba.otter.canal.parse.inbound.mysql.MysqlMultiStageCoprocessor$SinkStoreStage.onEvent(MysqlMultiStageCoprocessor.java:352) at com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:168) at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:748) server版本是1.1.4 @agapple 大佬帮忙看下 image

Rushing0711 commented 4 years ago

同问，canal的deployer服务，我这边线上发生过几次同步假死问题，看了gc正常，线程情况如提问中的截图，每次都发生在夜间2:30:18秒左右；查看了自己的业务代码，也没发现2:30干了啥，那会业务应该很低的。 暂时也没发现问题症结在哪里，期望大佬能查看下。

yourse007 commented 2 years ago

同问，canal的deployer服务，我这边线上发生过几次同步假死问题，看了gc正常，线程情况如提问中的截图，每次都发生在夜间2:30:18秒左右；查看了自己的业务代码，也没发现2:30干了啥，那会业务应该很低的。 暂时也没发现问题症结在哪里，期望大佬能查看下。

这个问题后续怎么样了？排查出来原因了吗