SaveSnapshot in ondisk example isn't called

qieziting commented 6 months ago

Dragonboat version

github.com/lni/dragonboat/v3 v3.3.8

Expected behavior

Call the following 3 function as expected:

SaveSnapshot(interface{}, io.Writer, <-chan struct{}) error
RecoverFromSnapshot(io.Reader, <-chan struct{}) error
PrepareSnapshot() (interface{}, error)

Actual behavior

Not called

Steps to reproduce the behavior

For I need to use v3 dragonboat in my project, so I changed the version of dragonboat in example to v3.3.8. Then I add some logger in the three fuction, such as:

func (d *DiskKV) PrepareSnapshot() (interface{}, error) {
    if d.closed {
        panic("prepare snapshot called after Close()")
    }
    if d.aborted {
        panic("prepare snapshot called after abort")
    }
    fmt.Println("Enter PrepareSnapshot")
    db := (*pebbledb)(atomic.LoadPointer(&d.db))
    return &diskKVCtx{
        db:       db,
        snapshot: db.db.NewSnapshot(),
    }, nil
}

func (d *DiskKV) SaveSnapshot(ctx interface{},
    w io.Writer, done <-chan struct{}) error {
    if d.closed {
        panic("prepare snapshot called after Close()")
    }
    if d.aborted {
        panic("prepare snapshot called after abort")
    }
    fmt.Println("Enter SaveSnapshot")
    ctxdata := ctx.(*diskKVCtx)
    db := ctxdata.db
    db.mu.RLock()
    defer db.mu.RUnlock()
    ss := ctxdata.snapshot
    defer ss.Close()
    return d.saveToWriter(db, ss, w)
}

Then I try to execute it, got following log:

node address: localhost:63001 2023-12-05 12:43:58.404989 I | dragonboat: go version: go1.19.4, linux/amd64 2023-12-05 12:43:58.405042 I | dragonboat: dragonboat version: 3.3.8 (Rel) 2023-12-05 12:43:58.405086 I | config: using default EngineConfig 2023-12-05 12:43:58.405096 I | config: using default LogDBConfig 2023-12-05 12:43:58.405132 I | dragonboat: DeploymentID set to 1 2023-12-05 12:43:58.412925 I | dragonboat: LogDB info received, shard 0, busy false 2023-12-05 12:43:58.418083 I | dragonboat: LogDB info received, shard 1, busy false 2023-12-05 12:43:58.422945 I | dragonboat: LogDB info received, shard 2, busy false 2023-12-05 12:43:58.427255 I | dragonboat: LogDB info received, shard 3, busy false 2023-12-05 12:43:58.431333 I | dragonboat: LogDB info received, shard 4, busy false 2023-12-05 12:43:58.435259 I | dragonboat: LogDB info received, shard 5, busy false 2023-12-05 12:43:58.439485 I | dragonboat: LogDB info received, shard 6, busy false 2023-12-05 12:43:58.444023 I | dragonboat: LogDB info received, shard 7, busy false 2023-12-05 12:43:58.448695 I | dragonboat: LogDB info received, shard 8, busy false 2023-12-05 12:43:58.452961 I | dragonboat: LogDB info received, shard 9, busy false 2023-12-05 12:43:58.458498 I | dragonboat: LogDB info received, shard 10, busy false 2023-12-05 12:43:58.462652 I | dragonboat: LogDB info received, shard 11, busy false 2023-12-05 12:43:58.469651 I | dragonboat: LogDB info received, shard 12, busy false 2023-12-05 12:43:58.474217 I | dragonboat: LogDB info received, shard 13, busy false 2023-12-05 12:43:58.478654 I | dragonboat: LogDB info received, shard 14, busy false 2023-12-05 12:43:58.483502 I | logdb: using plain logdb 2023-12-05 12:43:58.483656 I | dragonboat: LogDB info received, shard 15, busy false 2023-12-05 12:43:58.484702 I | dragonboat: logdb memory limit: 8192 MBytes 2023-12-05 12:43:58.485562 I | dragonboat: NodeHost ID: nhid-1214681883000945890 2023-12-05 12:43:58.485573 I | dragonboat: using regular node registry 2023-12-05 12:43:58.485577 I | dragonboat: filesystem error injection mode enabled: false 2023-12-05 12:43:58.487155 I | dragonboat: transport type: go-tcp-transport 2023-12-05 12:43:58.487172 I | dragonboat: logdb type: sharded-pebble 2023-12-05 12:43:58.487177 I | dragonboat: nodehost address: localhost:63001 2023-12-05 12:43:58.489278 I | dragonboat: [00128:00001] replaying raft logs Usage - put key value get key Enter open 2023-12-05 12:43:58.494901 I | dragonboat: [00128:00001] initialized using <00128:00001:0> 2023-12-05 12:43:58.494923 I | dragonboat: [00128:00001] initial index set to 0 put 1 1 Enter Update Update for index: 3 put 2 2 Enter Update Update for index: 4 put 3 3 Enter Update Update for index: 5 put 4 4 Enter Update Update for index: 6 put 5 5 Enter Update Update for index: 7 put 6 6 Enter Update Update for index: 8 put 7 7 Enter Update Update for index: 9 put 8 8 Enter Update Update for index: 10 put 9 9 Enter Update Update for index: 11 2023-12-05 12:44:23.524652 I | dragonboat: [00128:00001] saved snapshot, index <00128:00001:11>, term 2, file count 0 2023-12-05 12:44:23.689418 I | dragonboat: LogDB info received, shard 0, busy false 2023-12-05 12:44:23.690894 I | dragonboat: LogDB info received, shard 0, busy false 2023-12-05 12:44:23.691820 I | logdb: [00128:00001] completed LogDB compaction up to index 6 2023-12-05 12:44:23.691850 I | dragonboat: LogDB info received, shard 0, busy false put 10 10 Enter Update Update for index: 12

It looks like that logic of snapshot in dragonboat has been called, but PrepareSnapshotand SaveSnapshotaren't called. Could you check it? Thank you.

kevburnsjr commented 6 months ago

iirc, PrepareSnapshot and SaveSnapshot are only called during compaction for regular state machines and not for on-disk state machines. You should see a call to Sync instead. This makes sense because the replica's snapshot is already present on disk.

If you joined a new replica to the shard, you would likely see calls to PrepareSnapshot and SaveSnapshot to initialize the state of the new replica.

qieziting commented 6 months ago

Thanks for your answer.

If you joined a new replica to the shard, you would likely see calls to PrepareSnapshot and SaveSnapshot to initialize the state of the new replica.

I have try this one, and they are called.

Thank you very much.

lni / dragonboat