How to snapshot with existing snapshot files

[danthegoodman1]

When using something like SQLite in WAL mode to as the IOnDiskStateMachine storage backend, it should be possible to do something like https://philipotoole.com/building-rqlite-9-0-cutting-disk-usage-by-half/#:~:text=New%20Snapshotting%20approach where

1. The database file represents the most recent snapshot
2. The WAL is the persisted changes between UpdateEntires calls since the last snapshot
3. Snapshotting simply checkpoints

There does not seem to be the flexibility to provide custom snapshot file routing such that a snapshot call could checkpoint SQLite and instantly return, and a custom read snapshot method could stream the contents of the SQLite DB to a replica.

Is this possible with configuration to customize the snapshot file handling such that we can prevent duplicating the sqlite data on disk for snapshots?

I am using v4

[lni]

Hi @danthegoodman1

Based on my understand, everything you want is already in dragonboat since day one. Dragonboat doesn't duplicate state machine data by design.

When IOnDiskStateMachine is used, dragonboat doesn't periodically copy the entire content of your state machine as a part of dragonboat's own periodic snapshotting procedure. Dragonboat only saves its own tiny metadata as a part of such snapshotting procedure, it is thus instant by design.

When SaveSnapshot() is called, it does require you to copy the full content of your state machine (sqlite instance in your case), but that is for transporting/streaming it to a remote raft shard (e.g. a brand new shard that doesn't have anything on it) so it can restore from such provided state. The term streaming is used in code/comments as the sending side doesn't copy such data to its local disk first. The receiving end does store the received snapshot onto a local disk file first, it is to ensure that when RestoreFromSnapshot() is called, it is guaranteed to already have the full raft snapshot to save users from the trouble of handling partially damanged/lost raft snapshots.

Note that, the above mentioned SaveSnapshot() / RestoreFromSnapshot() are only called when a remote shard needs the full snapshot to catch up, they are not suppose to be invoked periodically.

[danthegoodman1]

Thank you for the clarification, so it only periodically calls PrepareSnapshot then?

[danthegoodman1]

Maybe not even? Looking at https://github.com/lni/dragonboat-example/blob/master/ondisk/diskkv.go#L424-L496 PrepareSnapshot is not returning anything that can be serialized.

[danthegoodman1]

It seems that for the regular snapshotting, it just serves as an indicator it's safe to truncate the log:

Applications that implement IOnDiskStateMachine are recommended to setup periodic snapshotting with relatively short time intervals, that triggers the state machine's metadata, usually only a few KBytes each, to be periodically snapshotted and thus causes negligible overheads for the system. It also provides opportunities for the system to signal Raft Log compactions to free up disk spaces.