lni / dragonboat

A feature complete and high performance multi-group Raft library in Go.
Apache License 2.0
5.08k stars 541 forks source link

ILogDB and Statemachine is sensitive to all kind of errors #191

Closed cloudlounger closed 2 years ago

cloudlounger commented 3 years ago

Dragonboat version

3.3.4

Expected behavior

when statemachine implement and ILogDB implement encounter errors, engine could use better graceful ways to handle errors, for example, the engine retry some work instead of panic immediately.

Actual behavior

when processSteps、processCommits、processApplies encounter errors, the process excute panicNow(err).

Steps to reproduce the behavior

when statemachie excute Update function, and the log entry applied but failed expected, it lead nodehost panic, and influence the other raft groups.

lni commented 2 years ago

when statemachie excute Update function, and the log entry applied but failed expected, it lead nodehost panic, and influence the other raft groups.

this issues have been discussed multiple times in the issue section.

when the log entry couldn't be applied because of some underlying errors (e.g. fail to write to disk), then panic is the right approach.

when the log entry couldn't be applied as the state of user state machine would reject such updates, then it is not an error to start with, it is just another condition that need to be handled by the state machine/clients. Rather than returning an error, please return some results indicating what happened.