search

lni / dragonboat

A feature complete and high performance multi-group Raft library in Go.
Apache License 2.0
5.09k stars 542 forks source link

Session support in IOnDiskStateMachine #356

Closed lni closed 6 months ago

lni commented 6 months ago

One of the dragonboat users recently asked why IOnDiskStateMachine doesn't support session. I'd like to provide the following details here so other potential users might find the info useful as well.

For sessions, they are internally managed by a state machine as well, all session data associated with user requests are eventually applied to that session state machine after they are committed. For in memory state machines, everything is pretty straightforward, dragonboat internally maintains a in-memory state machine with all session info, it is backed by a raft log and periodically snapshotted, everything is similar to how user state machine is maintained including recovery. As long as we apply both the session data and user request data at the same time, everything is fine.

For IOnDiskStateMachine, the underlying storage is persistent and dragonboat has no idea on what kind of storage is being used. To support sessions, there are basically two approaches -

  1. still maintain the session data as in memory state machine, this would lead to have both in memory and persistent state machines, it is obviously much more complicated as you need to be very careful during recovery to make sure everything is consistent.
  2. store session data in the persistent storage. the problem here is that dragonboat doesn't know anything about the storage while users shouldn't be required to know the details on how dragonboat works internally. Unfortunately, storing session data in user supplied storage would require dragonboat to make further assumptions on user provided storage or requiring users to know more about the session implementations.

It is thus decided not to support session - at the time of Dragonboat's public release I didn't even use IOnDiskStateMachine, everything is in-memory just like what the Raft thesis focused on. A later project of mine that did employ IOnDiskStateMachine allowed me to revisit this sessions and persistent storage issue. Eventually the session feature was handled entirely by the application as a part of the IOnDiskStateMachine implementation.

At this stage, there is no plan to add session support for IOnDiskStateMachine as it would be pretty hard to verify the implementation to make sure it is solid reliable. If required, users can always roll their own implementation as part of the IOnDiskStateMachine.

kevburnsjr commented 6 months ago

Relevant Architecture Decision Record from one if my projects using Dragonboat: https://github.com/logbn/zongzi/blob/main/docs/adr/raft_sessions.md.

The only advantage I can see to implementing sessions as a part of the protocol rather than delegating this to the state machine is to avoid committing duplicate proposals. If deduplication is implemented by the state machine, some duplicate proposals may be committed before being discarded.

In practice, this overhead appears to be trivial. I have failed to imagine any pathological edge case that couldn't be solved by thoughtful design.

I suspect that sessions were only included in the raft thesis and are only required in cases where you don't have direct control over the state machine to implement deduplication of committed proposals. In that case, the protocol can take some of the burden which is at least a nice feature to have for the sake of simplicity.

For anyone who wants to implement their own sessions for an IOnDiskStateMachine, TigerBeetle has good Client Sessions documentation to start your design process.