search

sofastack / sofa-jraft

A production-grade java implementation of RAFT consensus algorithm.
https://www.sofastack.tech/projects/sofa-jraft/
Apache License 2.0
3.58k stars 1.15k forks source link

Inserting 20,000 data in batches and starting the error cluster hanging #297

Closed zhangjun050754 closed 4 years ago

zhangjun050754 commented 4 years ago

10-15 22:57:51.273 ERROR [com.alipay.sofa.jraft.util.LogExceptionHandler] - Handle LogManagerImpl disruptor event error, event is com.alipay.sofa.jraft.storage.impl.LogManagerImpl$StableClosureEvent@68bbb78c java.lang.NullPointerException: null at com.alipay.sofa.jraft.storage.impl.LogManagerImpl$StableClosureEventHandler.onEvent(LogManagerImpl.java:518) at com.alipay.sofa.jraft.storage.impl.LogManagerImpl$StableClosureEventHandler.onEvent(LogManagerImpl.java:494) at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:137) at java.lang.Thread.run(Thread.java:748) 10-15 22:57:51.273 WARN [com.alipay.sofa.jraft.core.FSMCallerImpl] - FSMCaller already in error status, ignore new error: {} com.alipay.sofa.jraft.error.RaftException: ERROR_TYPE_LOG at com.alipay.sofa.jraft.storage.impl.LogManagerImpl.reportError(LogManagerImpl.java:572) at com.alipay.sofa.jraft.storage.impl.LogManagerImpl.lambda$init$0(LogManagerImpl.java:205) at com.alipay.sofa.jraft.util.LogExceptionHandler.handleEventException(LogExceptionHandler.java:67) at com.lmax.disruptor.dsl.ExceptionHandlerWrapper.handleEventException(ExceptionHandlerWrapper.java:18) at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:156) at java.lang.Thread.run(Thread.java:748)

killme2008 commented 4 years ago

这不是最原始的错误了，状态机已经进入 error 状态了，你可以找下最开始的错误，一般会有提示信息。

killme2008 commented 4 years ago

另外请提供下版本信息等

zhangjun050754 commented 4 years ago

2019-10-15 22:57:36,761 [JRaft-Closure-Executor-99] ERROR [com.alipay.sofa.jraft.core.NodeImpl] NodeImpl.java:1073 - Node <rhea_mqtt--1/127.0.0.1:7181> append [0, 0] failed. 2019-10-15 22:57:36,761 [JRaft-Closure-Executor-98] ERROR [com.alipay.sofa.jraft.core.NodeImpl] NodeImpl.java:1073 - Node <rhea_mqtt--1/127.0.0.1:7181> append [0, 0] failed. 2019-10-15 22:57:36,760 [JRaft-Closure-Executor-94] ERROR [com.alipay.sofa.jraft.core.NodeImpl] NodeImpl.java:1073 - Node <rhea_mqtt--1/127.0.0.1:7181> append [0, 0] failed. 2019-10-15 22:57:36,761 [JRaft-Closure-Executor-97] ERROR [com.alipay.sofa.jraft.core.NodeImpl] NodeImpl.java:1073 - Node <rhea_mqtt--1/127.0.0.1:7181> append [0, 0] failed. 2019-10-15 22:57:36,760 [JRaft-Closure-Executor-95] ERROR [com.alipay.sofa.jraft.core.NodeImpl] NodeImpl.java:1073 - Node <rhea_mqtt--1/127.0.0.1:7181> append [0, 0] failed. 2019-10-15 22:57:36,760 [JRaft-Closure-Executor-96] ERROR [com.alipay.sofa.jraft.core.NodeImpl] NodeImpl.java:1073 - Node <rhea_mqtt--1/127.0.0.1:7181> append [0, 0] failed. 2019-10-15 22:57:36,760 [JRaft-Closure-Executor-93] ERROR [com.alipay.sofa.jraft.core.NodeImpl] NodeImpl.java:1073 - Node <rhea_mqtt--1/127.0.0.1:7181> append [0, 0] failed. 2019-10-15 22:57:36,759 [JRaft-Closure-Executor-92] ERROR [com.alipay.sofa.jraft.core.NodeImpl] NodeImpl.java:1073 - Node <rhea_mqtt--1/127.0.0.1:7181> append [0, 0] failed. 2019-10-15 22:57:36,759 [JRaft-Closure-Executor-91] ERROR [com.alipay.sofa.jraft.core.NodeImpl] NodeImpl.java:1073 - Node <rhea_mqtt--1/127.0.0.1:7181> append [0, 0] failed. 2019-10-15 22:57:36,759 [JRaft-Closure-Executor-90] ERROR [com.alipay.sofa.jraft.core.NodeImpl] NodeImpl.java:1073 - Node <rhea_mqtt--1/127.0.0.1:7181> append [0, 0] failed. 2019-10-15 22:57:36,759 [JRaft-Closure-Executor-89] ERROR [com.alipay.sofa.jraft.core.NodeImpl] NodeImpl.java:1073 - Node <rhea_mqtt--1/127.0.0.1:7181> append [0, 0] failed. 2019-10-15 22:57:36,759 [JRaft-Closure-Executor-86] ERROR [com.alipay.sofa.jraft.core.NodeImpl] NodeImpl.java:1073 - Node <rhea_mqtt--1/127.0.0.1:7181> append [0, 0] failed. 2019-10-15 22:57:36,759 [JRaft-Closure-Executor-88] ERROR [com.alipay.sofa.jraft.core.NodeImpl] NodeImpl.java:1073 - Node <rhea_mqtt--1/127.0.0.1:7181> append [0, 0] failed. 2019-10-15 22:57:36,759 [JRaft-Closure-Executor-87] ERROR [com.alipay.sofa.jraft.core.NodeImpl] NodeImpl.java:1073 - Node <rhea_mqtt--1/127.0.0.1:7181> append [0, 1] failed. 2019-10-15 22:57:36,758 [JRaft-Closure-Executor-82] ERROR [com.alipay.sofa.jraft.core.NodeImpl] NodeImpl.java:1073 - Node <rhea_mqtt--1/127.0.0.1:7181> append [0, 0] failed. 2019-10-15 22:57:36,757 [JRaft-Closure-Executor-79] ERROR [com.alipay.sofa.jraft.core.NodeImpl] NodeImpl.java:1073 - Node <rhea_mqtt--1/127.0.0.1:7181> append [0, 0] failed. 2019-10-15 22:57:36,758 [JRaft-Closure-Executor-85] ERROR [com.alipay.sofa.jraft.core.NodeImpl] NodeImpl.java:1073 - Node <rhea_mqtt--1/127.0.0.1:7181> append [0, 0] failed. 2019-10-15 22:57:36,758 [JRaft-Closure-Executor-84] ERROR [com.alipay.sofa.jraft.core.NodeImpl] NodeImpl.java:1073 - Node <rhea_mqtt--1/127.0.0.1:7181> append [0, 0] failed. 2019-10-15 22:57:36,826 [JRaft-NodeImpl-Disruptor-0] ERROR [com.alipay.sofa.jraft.util.LogExceptionHandler] LogExceptionHandler.java:65 - Handle NodeImpl disruptor event error, event is com.alipay.sofa.jraft.core.NodeImpl$LogEntryAndClosure@4704a3b java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@38ec2fa3 rejected from com.alipay.sofa.jraft.util.MetricThreadPoolExecutor@4ff18d96[Running, pool size = 100, active threads = 99, queued tasks = 0, completed tasks = 7680] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2047) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1369) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at com.alipay.sofa.jraft.util.Utils.runInThread(Utils.java:153) at com.alipay.sofa.jraft.util.Utils.runClosureInThread(Utils.java:163) at com.alipay.sofa.jraft.storage.impl.LogManagerImpl.appendEntries(LogManagerImpl.java:301) at com.alipay.sofa.jraft.core.NodeImpl.executeApplyingTasks(NodeImpl.java:1119) at com.alipay.sofa.jraft.core.NodeImpl.access$200(NodeImpl.java:129) at com.alipay.sofa.jraft.core.NodeImpl$LogEntryAndClosureHandler.onEvent(NodeImpl.java:259) at com.alipay.sofa.jraft.core.NodeImpl$LogEntryAndClosureHandler.onEvent(NodeImpl.java:240) at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:137) at java.lang.Thread.run(Thread.java:748) 2019-10-15 22:57:36,757 [JRaft-Closure-Executor-81] ERROR [com.alipay.sofa.jraft.core.NodeImpl] NodeImpl.java:1073 - Node <rhea_mqtt--1/127.0.0.1:7181> append [0, 0] failed. 2019-10-15 22:57:36,757 [JRaft-Closure-Executor-77] ERROR [com.alipay.sofa.jraft.core.NodeImpl] NodeImpl.java:1073 - Node <rhea_mqtt--1/127.0.0.1:7181> append [0, 0] failed. 2019-10-15 22:57:36,756 [JRaft-Closure-Executor-73] ERROR [com.alipay.sofa.jraft.core.NodeImpl] NodeImpl.java:1073 - Node <rhea_mqtt--1/127.0.0.1:7181> append [0, 0] failed. 2019-10-15 22:57:36,756 [JRaft-Closure-Executor-71] ERROR [com.alipay.sofa.jraft.core.NodeImpl] NodeImpl.java:1073 - Node <rhea_mqtt--1/127.0.0.1:7181> append [0, 0] failed. 2019-10-15 22:57:36,757 [JRaft-Closure-Executor-78] ERROR [com.alipay.sofa.jraft.core.NodeImpl] NodeImpl.java:1073 - Node <rhea_mqtt--1/127.0.0.1:7181> append [0, 0] failed. 2019-10-15 22:57:36,826 [JRaft-Closure-Executor-84] ERROR [com.alipay.sofa.jraft.core.NodeImpl] NodeImpl.java:1073 - Node <rhea_mqtt--1/127.0.0.1:7181> append [0, 32] failed. 2019-10-15 22:57:36,757 [JRaft-Closure-Executor-80] ERROR [com.alipay.sofa.jraft.core.NodeImpl] NodeImpl.java:1073 - Node <rhea_mqtt--1/127.0.0.1:7181> append [0, 0] failed. 2019-10-15 22:57:36,826 [JRaft-Closure-Executor-85] ERROR [com.alipay.sofa.jraft.core.NodeImpl] NodeImpl.java:1073 - Node <rhea_mqtt--1/127.0.0.1:7181> append [0, 31] failed. 2019-10-15 22:57:36,753 [JRaft-Closure-Executor-58] ERROR [com.alipay.sofa.jraft.core.NodeImpl] NodeImpl.java:1073 - Node <rhea_mqtt--1/127.0.0.1:7181> append [0, 0] failed. 2019-10-15 22:57:36,754 [JRaft-Closure-Executor-64] ERROR [com.alipay.sofa.jraft.core.NodeImpl] NodeImpl.java:1073 - Node <rhea_mqtt--1/127.0.0.1:7181> append [0, 0] failed. 2019-10-15 22:57:36,758 [JRaft-Closure-Executor-83] ERROR [com.alipay.sofa.jraft.core.NodeImpl] NodeImpl.java:1073 - Node <rhea_mqtt--1/127.0.0.1:7181> append [0, 0] failed. 2019-10-15 22:57:36,756 [JRaft-Closure-Executor-75] ERROR [com.alipay.sofa.jraft.core.NodeImpl] NodeImpl.java:1073 - Node <rhea_mqtt--1/127.0.0.1:7181> append [0, 0] failed. 2019-10-15 22:57:36,753 [JRaft-Closure-Executor-56] ERROR [com.alipay.sofa.jraft.core.NodeImpl] NodeImpl.java:1073 - Node <rhea_mqtt--1/127.0.0.1:7181> append [0, 0] failed. 2019-10-15 22:57:36,826 [JRaft-Closure-Executor-73] ERROR [com.alipay.sofa.jraft.core.NodeImpl] NodeImpl.java:1073 - Node <rhea_mqtt--1/127.0.0.1:7181> append [0, 31] failed. 2019-10-15 22:57:36,756 [JRaft-Closure-Executor-76] ERROR [com.alipay.sofa.jraft.core.NodeImpl] NodeImpl.java:1073 - Node <rhea_mqtt--1/127.0.0.1:7181> append [0, 0] failed. 2019-10-15 22:57:36,756 [JRaft-Closure-Executor-74] ERROR [com.alipay.sofa.jraft.core.NodeImpl] NodeImpl.java:1073 - Node <rhea_mqtt--1/127.0.0.1:7181> append [0, 1] failed. 2019-10-15 22:57:36,756 [JRaft-Closure-Executor-72] ERROR [com.alipay.sofa.jraft.core.NodeImpl] NodeImpl.java:1073 - Node <rhea_mqtt--1/127.0.0.1:7181> append [0, 0] failed. 2019-10-15 22:57:36,755 [JRaft-Closure-Executor-70] ERROR [com.alipay.sofa.jraft.core.NodeImpl] NodeImpl.java:1073 - Node <rhea_mqtt--1/127.0.0.1:7181> append [0, 0] failed. 2019-10-15 22:57:36,751 [JRaft-Closure-Executor-48] ERROR [com.alipay.sofa.jraft.core.NodeImpl] NodeImpl.java:1073 - Node <rhea_mqtt--1/127.0.0.1:7181> append [0, 0] failed. 2019-10-15 22:57:36,752 [JRaft-Closure-Executor-50] ERROR [com.alipay.sofa.jraft.core.NodeImpl] NodeImpl.java:1073 - Node <rhea_mqtt--1/127.0.0.1:7181> append [0, 0] failed. 2019-10-15 22:57:36,750 [JRaft-Closure-Executor-38] ERROR [com.alipay.sofa.jraft.core.NodeImpl] NodeImpl.java:1073 - Node <rhea_mqtt--1/127.0.0.1:7181> append [0, 0] failed. 2019-10-15 22:57:36,755 [JRaft-Closure-Executor-68] ERROR [com.alipay.sofa.jraft.core.NodeImpl] NodeImpl.java:1073 - Node <rhea_mqtt--1/127.0.0.1:7181> append [0, 0] failed. 2019-10-15 22:57:36,751 [JRaft-Closure-Executor-43] ERROR [com.alipay.sofa.jraft.core.NodeImpl] NodeImpl.java:1073 - Node <rhea_mqtt--1/127.0.0.1:7181> append [0, 0] failed. 2019-10-15 22:57:36,755 [JRaft-Closure-Executor-69] ERROR [com.alipay.sofa.jraft.core.NodeImpl] NodeImpl.java:1073 - Node <rhea_mqtt--1/127.0.0.1:7181> append [0, 0] failed. 2019-10-15 22:57:36,755 [JRaft-Closure-Executor-67] ERROR [com.alipay.sofa.jraft.core.NodeImpl] NodeImpl.java:1073 - Node <rhea_mqtt--1/127.0.0.1:7181> append [0, 0] failed. 2019-10-15 22:57:36,755 [JRaft-Closure-Executor-66] ERROR [com.alipay.sofa.jraft.core.NodeImpl] NodeImpl.java:1073 - Node <rhea_mqtt--1/127.0.0.1:7181> append [0, 0] failed. 2019-10-15 22:57:36,754 [JRaft-Closure-Executor-65] ERROR [com.alipay.sofa.jraft.core.NodeImpl] NodeImpl.java:1073 - Node <rhea_mqtt--1/127.0.0.1:7181> append [0, 0] failed. 2019-10-15 22:57:36,754 [JRaft-Closure-Executor-63] ERROR [com.alipay.sofa.jraft.core.NodeImpl] NodeImpl.java:1073 - Node <rhea_mqtt--1/127.0.0.1:7181> append [0, 1] failed. 2019-10-15 22:57:36,826 [JRaft-Closure-Executor-73] ERROR [com.alipay.sofa.jraft.core.NodeImpl] NodeImpl.java:1073 - Node <rhea_mqtt--1/127.0.0.1:7181> append [0, 0] failed. 2019-10-15 22:57:36,746 [JRaft-Closure-Executor-19] ERROR [com.alipay.sofa.jraft.core.NodeImpl] NodeImpl.java:1073 - Node <rhea_mqtt--1/127.0.0.1:7181> append [0, 0] failed. 2019-10-15 22:57:36,746 [JRaft-Closure-Executor-16] ERROR [com.alipay.sofa.jraft.core.NodeImpl] NodeImpl.java:1073 - Node <rhea_mqtt--1/127.0.0.1:7181> append [0, 0] failed. 2019-10-15 22:57:36,754 [JRaft-Closure-Executor-62] ERROR [com.alipay.sofa.jraft.core.NodeImpl] NodeImpl.java:1073 - Node <rhea_mqtt--1/127.0.0.1:7181> append [0, 0] failed. 2019-10-15 22:57:36,745 [JRaft-Closure-Executor-3] ERROR [com.alipay.sofa.jraft.core.NodeImpl] NodeImpl.java:1073 - Node <rhea_mqtt--1/127.0.0.1:7181> append [0, 0] failed. 2019-10-15 22:57:36,828 [JRaft-Closure-Executor-69] ERROR [com.alipay.sofa.jraft.core.NodeImpl] NodeImpl.java:1073 - Node <rhea_mqtt--1/127.0.0.1:7181> append [0, 1] failed. 2019-10-15 22:57:36,826 [JRaft-Closure-Executor-81] ERROR [com.alipay.sofa.jraft.core.NodeImpl] NodeImpl.java:1073 - Node <rhea_mqtt--1/127.0.0.1:7181> append [0, 31] failed. 2019-10-15 22:57:36,753 [JRaft-Closure-Executor-60] ERROR [com.alipay.sofa.jraft.core.NodeImpl] NodeImpl.java:1073 - Node <rhea_mqtt--1/127.0.0.1:7181> append [0, 0] failed. 2019-10-15 22:57:36,751 [JRaft-Closure-Executor-47] ERROR [com.alipay.sofa.jraft.core.NodeImpl] NodeImpl.java:1073 - Node <rhea_mqtt--1/127.0.0.1:7181> append [0, 0] failed. 2019-10-15 22:57:36,742 [JRaft-Closure-Executor-15] ERROR [com.alipay.sofa.jraft.core.NodeImpl] NodeImpl.java:1073 - Node <rhea_mqtt--1/127.0.0.1:7181> append [0, 0] failed. 2019-10-15 22:57:36,826 [JRaft-Closure-Executor-87] ERROR [com.alipay.sofa.jraft.core.NodeImpl] NodeImpl.java:1073 - Node <rhea_mqtt--1/127.0.0.1:7181> append [0, 31] failed. 2019-10-15 22:57:36,826 [JRaft-Closure-Executor-82] ERROR [com.alipay.sofa.jraft.core.NodeImpl] NodeImpl.java:1073 - Node <rhea_mqtt--1/127.0.0.1:7181> append [0, 31] failed. 2019-10-15 22:57:36,743 [JRaft-Closure-Executor-10] ERROR [com.alipay.sofa.jraft.core.NodeImpl] NodeImpl.java:1073 - Node <rhea_mqtt--1/127.0.0.1:7181> append [0, 0] failed. 2019-10-15 22:57:36,744 [JRaft-Closure-Executor-4] ERROR [com.alipay.sofa.jraft.core.NodeImpl] NodeImpl.java:1073 - Node <rhea_mqtt--1/127.0.0.1:7181> append [0, 0] failed. 2019-10-15 22:57:36,752 [JRaft-Closure-Executor-55] ERROR [com.alipay.sofa.jraft.core.NodeImpl] NodeImpl.java:1073 - Node <rhea_mqtt--1/127.0.0.1:7181> append [0, 0] failed. 2019-10-15 22:57:36,745 [JRaft-Closure-Executor-2] ERROR [com.alipay.sofa.jraft.core.NodeImpl] NodeImpl.java:1073 - Node <rhea_mqtt--1/127.0.0.1:7181> append [0, 0] failed. 2019-10-15 22:57:36,746 [JRaft-Closure-Executor-18] ERROR [com.alipay.sofa.jraft.core.NodeImpl] NodeImpl.java:1073 - Node <rhea_mqtt--1/127.0.0.1:7181> append [0, 0] failed. 2019-10-15 22:57:36,752 [JRaft-Closure-Executor-53] ERROR [com.alipay.sofa.jraft.core.NodeImpl] NodeImpl.java:1073 - Node <rhea_mqtt--1/127.0.0.1:7181> append [0, 0] failed. 2019-10-15 22:57:36,747 [JRaft-Closure-Executor-24] ERROR [com.alipay.sofa.jraft.core.NodeImpl] NodeImpl.java:1073 - Node <rhea_mqtt--1/127.0.0.1:7181> append [0, 0] failed. 2019-10-15 22:57:36,746 [JRaft-Closure-Executor-22] ERROR [com.alipay.sofa.jraft.core.NodeImpl] NodeImpl.java:1073 - Node <rhea_mqtt--1/127.0.0.1:7181> append [0, 0] failed. 2019-10-15 22:57:36,746 [JRaft-Closure-Executor-23] ERROR [com.alipay.sofa.jraft.core.NodeImpl] NodeImpl.java:1073 - Node <rhea_mqtt--1/127.0.0.1:7181> append [0, 0] failed. 2019-10-15 22:57:36,752 [JRaft-Closure-Executor-52] ERROR [com.alipay.sofa.jraft.core.NodeImpl] NodeImpl.java:1073 - Node <rhea_mqtt--1/127.0.0.1:7181> append [0, 0] failed. 2019-10-15 22:57:36,752 [JRaft-Closure-Executor-54] ERROR [com.alipay.sofa.jraft.core.NodeImpl] NodeImpl.java:1073 - Node <rhea_mqtt--1/127.0.0.1:7181> append [0, 0] failed. 2019-10-15 22:57:36,826 [JRaft-Closure-Executor-79] ERROR [com.alipay.sofa.jraft.core.NodeImpl] NodeImpl.java:1073 - Node <rhea_mqtt--1/127.0.0.1:7181> append [0, 31] failed. 2019-10-15 22:57:36,753 [JRaft-Closure-Executor-57] ERROR [com.alipay.sofa.jraft.core.NodeImpl] NodeImpl.java:1073 - Node <rhea_mqtt--1/127.0.0.1:7181> append [0, 1] failed. 2019-10-15 22:57:36,753 [JRaft-Closure-Executor-59] ERROR [com.alipay.sofa.jraft.core.NodeImpl] NodeImpl.java:1073 - Node <rhea_mqtt--1/127.0.0.1:7181> append [0, 0] failed. 2019-10-15 22:57:36,826 [JRaft-Closure-Executor-77] ERROR [com.alipay.sofa.jraft.core.NodeImpl] NodeImpl.java:1073 - Node <rhea_mqtt--1/127.0.0.1:7181> append [0, 31] failed. 2019-10-15 22:57:36,754 [JRaft-Closure-Executor-61] ERROR [com.alipay.sofa.jraft.core.NodeImpl] NodeImpl.java:1073 - Node <rhea_mqtt--1/127.0.0.1:7181> append [0, 0] failed. 2019-10-15 22:57:36,752 [JRaft-Closure-Executor-51] ERROR [com.alipay.sofa.jraft.core.NodeImpl] NodeImpl.java:1073 - Node <rhea_mqtt--1/127.0.0.1:7181> append [0, 0] failed. 2019-10-15 22:57:36,752 [JRaft-Closure-Executor-49] ERROR [com.alipay.sofa.jraft.core.NodeImpl] NodeImpl.java:1073 - Node <rhea_mqtt--1/127.0.0.1:7181> append [0, 0] failed. 2019-10-15 22:57:36,751 [JRaft-Closure-Executor-45] ERROR [com.alipay.sofa.jraft.core.NodeImpl] NodeImpl.java:1073 - Node <rhea_mqtt--1/127.0.0.1:7181> append [0, 0] failed. 2019-10-15 22:57:36,750 [JRaft-Closure-Executor-41] ERROR [com.alipay.sofa.jraft.core.NodeImpl] NodeImpl.java:1073 - Node <rhea_mqtt--1/127.0.0.1:7181> append [0, 0] failed. 2019-10-15 22:57:36,750 [JRaft-Closure-Executor-39] ERROR [com.alipay.sofa.jraft.core.NodeImpl] NodeImpl.java:1073 - Node <rhea_mqtt--1/127.0.0.1:7181> append [0, 0] failed. 2019-10-15 22:57:36,750 [JRaft-Closure-Executor-40] ERROR [com.alipay.sofa.jraft.core.NodeImpl] NodeImpl.java:1073 - Node <rhea_mqtt--1/127.0.0.1:7181> append [0, 0] failed. 2019-10-15 22:57:36,750 [JRaft-Closure-Executor-42] ERROR [com.alipay.sofa.jraft.core.NodeImpl] NodeImpl.java:1073 - Node <rhea_mqtt--1/127.0.0.1:7181> append [0, 0] failed. 2019-10-15 22:57:36,749 [JRaft-Closure-Executor-37] ERROR [com.alipay.sofa.jraft.core.NodeImpl] NodeImpl.java:1073 - Node <rhea_mqtt--1/127.0.0.1:7181> append [0, 0] failed. 2019-10-15 22:57:36,749 [JRaft-Closure-Executor-35] ERROR [com.alipay.sofa.jraft.core.NodeImpl] NodeImpl.java:1073 - Node <rhea_mqtt--1/127.0.0.1:7181> append [0, 1] failed. 2019-10-15 22:57:36,749 [JRaft-Closure-Executor-36] ERROR [com.alipay.sofa.jraft.core.NodeImpl] NodeImpl.java:1073 - Node <rhea_mqtt--1/127.0.0.1:7181> append [0, 0] failed. 2019-10-15 22:57:36,749 [JRaft-Closure-Executor-34] ERROR [com.alipay.sofa.jraft.core.NodeImpl] NodeImpl.java:1073 - Node <rhea_mqtt--1/127.0.0.1:7181> append [0, 0] failed. 2019-10-15 22:57:36,748 [JRaft-Closure-Executor-29] ERROR [com.alipay.sofa.jraft.core.NodeImpl] NodeImpl.java:1073 - Node <rhea_mqtt--1/127.0.0.1:7181> append [0, 0] failed. 2019-10-15 22:57:36,748 [JRaft-Closure-Executor-30] ERROR [com.alipay.sofa.jraft.core.NodeImpl] NodeImpl.java:1073 - Node <rhea_mqtt--1/127.0.0.1:7181> append [0, 1] failed. 2019-10-15 22:57:36,748 [JRaft-Closure-Executor-31] ERROR [com.alipay.sofa.jraft.core.NodeImpl] NodeImpl.java:1073 - Node <rhea_mqtt--1/127.0.0.1:7181> append [0, 0] failed. 2019-10-15 22:57:36,751 [JRaft-Closure-Executor-44] ERROR [com.alipay.sofa.jraft.core.NodeImpl] NodeImpl.java:1073 - Node <rhea_mqtt--1/127.0.0.1:7181> append [0, 0] failed. 2019-10-15 22:57:36,747 [JRaft-Closure-Executor-26] ERROR [com.alipay.sofa.jraft.core.NodeImpl] NodeImpl.java:1073 - Node <rhea_mqtt--1/127.0.0.1:7181> append [0, 0] failed. 2019-10-15 22:57:36,747 [JRaft-Closure-Executor-25] ERROR [com.alipay.sofa.jraft.core.NodeImpl] NodeImpl.java:1073 - Node <rhea_mqtt--1/127.0.0.1:7181> append [0, 1] failed. 2019-10-15 22:57:36,828 [JRaft-Closure-Executor-43] ERROR [com.alipay.sofa.jraft.core.NodeImpl] NodeImpl.java:1073 - Node <rhea_mqtt--1/127.0.0.1:7181> append [0, 0] failed. 2019-10-15 22:57:36,745 [JRaft-Closure-Executor-0] ERROR [com.alipay.sofa.jraft.core.NodeImpl] NodeImpl.java:1073 - Node <rhea_mqtt--1/127.0.0.1:7181> append [0, 0] failed. 2019-10-15 22:57:36,745 [JRaft-Closure-Executor-1] ERROR [com.alipay.sofa.jraft.core.NodeImpl] NodeImpl.java:1073 - Node <rhea_mqtt--1/127.0.0.1:7181> append [0, 0] failed. 2019-10-15 22:57:36,751 [JRaft-Closure-Executor-46] ERROR [com.alipay.sofa.jraft.core.NodeImpl] NodeImpl.java:1073 - Node <rhea_mqtt--1/127.0.0.1:7181> append [0, 0] failed. 2019-10-15 22:57:36,745 [JRaft-FSMCaller-Disruptor-0] ERROR [com.alipay.sofa.jraft.core.StateMachineAdapter] StateMachineAdapter.java:72 - Encountered an error=Status[EBUSY<1009>: LogManager is busy, disk queue overload.] on StateMachine com.alipay.sofa.jraft.rhea.storage.KVStoreStateMachine, it's highly recommended to implement this method as raft stops working since some error ocurrs, you should figure out the cause and repair or remove this node.

zhangjun050754 commented 4 years ago 
    <!-- jraft -->
    <dependency>
        <groupId>com.alipay.sofa</groupId>
        <artifactId>jraft-rheakv-core</artifactId>
        <version>1.2.6</version>
    </dependency>
killme2008 commented 4 years ago

过载保护了 LogManager is busy, disk queue overload. ，你的请求应该都是异步批量提交的吧，没有同步等待写入。

这种情况下可以通过增大 RaftOptions#setDisruptorBufferSize() 来解决，默认是 16384。不过如果写入量仍然过大，可能会撑爆内存。

killme2008 commented 4 years ago

注意这行错误

Encountered an error=Status[EBUSY<1009>: LogManager is busy, disk queue overload.] on StateMachine com.alipay.sofa.jraft.rhea.storage.KVStoreStateMachine, it's highly recommended to implement this method as raft stops working since some error ocurrs, you should figure out the cause and repair or remove this node.
zhangjun050754 commented 4 years ago

注意这行错误

Encountered an error=Status[EBUSY<1009>: LogManager is busy, disk queue overload.] on StateMachine com.alipay.sofa.jraft.rhea.storage.KVStoreStateMachine, it's highly recommended to implement this method as raft stops working since some error ocurrs, you should figure out the cause and repair or remove this node.

好的 感谢支持 确实用的异步put提交的 主要做个for循环测一下性能

killme2008 commented 4 years ago

@zhangjun050754 客气。单个 raft group 的性能是有上限的，取决于很多因素，包括你的硬件配置、 状态机实现、使用方式以及一些参数等，更多调整参数参见 https://www.sofastack.tech/projects/sofa-jraft/jraft-user-guide/ 这里的 2.3 节，以及 8.2 节等。

如果单个 raft group 无法满足性能，推荐使用多个 raft group，业务上做分片。

fengjiachun commented 4 years ago

没其他问题先关闭了