Any possiblity to apply last applied index at server restart?

[mareden]

Your question

As described in JRaft documenth, once server is restarted, snapshot and log entries will be replayed and applied on the state machine. In case the state machine data is persisted, is it possible to persist the 'lastAppliedIndex' (or a recent one if it's hard to persist real last one) so that that index can be used at state machine restart step, so that duplicated log/snapshot process could be avoided?

Your scenes

For 'persisted state machine' case, once the log engines are successfully applied, the result is persisted and available after restart. So the question is about the 'persisted state machine'.

Your advice

It's suggested to provide some way to specify a 'last applied index'. Actually, the similar idea is described here: https://stackoverflow.com/a/53811985

Environment

• SOFAJRaft version:1.3.4
• JVM version (e.g. java -version): JDK1.8
• OS version (e.g. uname -a):
• Maven version:
• IDE version:

[fengjiachun]

com.alipay.sofa.jraft.FSMCaller#addLastAppliedLogIndexListener ?

[mareden]

com.alipay.sofa.jraft.FSMCaller#addLastAppliedLogIndexListener ?

Thanks for quick response. Actually this question is related to both how to get 'last applied index' and how to apply last applied index after restart. I believe that possibly FSMCaller#addLastAppliedLogIndexListener may provide a way to retrieve last applied index (and inside StateMachine.onApply, Iterator parameter can provide last index) which could be retrieved and thus persisted. On the other side, it is still unclear how to apply this lastAppliedIndex after state machine restart. I've checked, and found that FSMCaller is a private member for Node (actually NdoeImpl) class, and it's not easy to set/config last applied index after state machine restart.

[mareden]

After looking into JRaft source code, I have some findings, though not knowing if they can achieve the purpose.

One is NodeImpl.bootstrap(BootstrapOptions), and another is JRaftUtils.bootstrap(BootstrapOptions). Both have the code to call 'opts.setBootstrapId(bootstrapId)' to set bootstrapId/lastAppliedIndex.

Any idea to use the mentioned methods (or write similar code) to set the index after node restart?

Currently, the node startup code looks like: final RpcServer rpcServer = RaftRpcServerFactory.createRaftRpcServer(serverId.getEndpoint()); this.raftGroupService = new RaftGroupService("test_group, serverId, nodeOptions, rpcServer); this.node = this.raftGroupService.start();

This means the node object is from RaftGroupService.start(), which calls RaftServiceFactory.createAndInitRaftNode() and internally sets LogId(0, 0), a hard-coded value. So if there's any idea to specify the log id, we'll get expected result.

@fengjiachun It'll be really helpful if you can share your thoughts. Thanks.

[killme2008]

You can persistent the lastAppliedIndex by your self in StateMachine#onApply(Iterator). Retrieve the value by iter.getIndex()-1, and save it in persist storage such as a file or database etc.

If you are using snapshot, you can retrieve the snapshot meta by SnapshotReader#load() in StateMachine#onSnapshotLoad(SnapshotReader), the meta instance contains the lastIncludeIndex and lastIncludeTerm of the snapshot.