When stopping a stream or consumer, we would attempt to install a snapshot. However, this would race with what's happening in monitorStream/monitorConsumer at that time.
For example:
In applyStreamEntries we call into mset.processJetStreamMsg to persist one or multiple messages.
We call mset.stop(..) either before or during the above.
In mset.stop(..) we'd wait for mset.processJetStreamMsg to release the lock so we can enter mset.stateSnapshotLocked(). We create a snapshot with new state here!
Now we call into InstallSnapshot to persist above snapshot, but n.applied does not contain the right value, the value will be lower.
Then applyStreamEntries finishes and we end with calling n.Applied(..).
This would be a race condition depending on if 4 happened before or after 5.
It's essential that the snapshot we make is aligned with the n.applied value. If we don't that means we'll replay and need to increase mset.clfs which will snowball into stream desync due to this shift.
The only place where we can guarantee that the snapshot and applied are aligned is in doSnapshot of monitorStream and monitorConsumer (and monitorCluster), so we must not attempt installing snapshots outside of those.
When stopping a stream or consumer, we would attempt to install a snapshot. However, this would race with what's happening in
monitorStream/monitorConsumerat that time.For example:
applyStreamEntrieswe call intomset.processJetStreamMsgto persist one or multiple messages.mset.stop(..)either before or during the above.mset.stop(..)we'd wait formset.processJetStreamMsgto release the lock so we can entermset.stateSnapshotLocked(). We create a snapshot with new state here!InstallSnapshotto persist above snapshot, butn.applieddoes not contain the right value, the value will be lower.applyStreamEntriesfinishes and we end with callingn.Applied(..).This would be a race condition depending on if 4 happened before or after 5.
It's essential that the snapshot we make is aligned with the
n.appliedvalue. If we don't that means we'll replay and need to increasemset.clfswhich will snowball into stream desync due to this shift.The only place where we can guarantee that the snapshot and applied are aligned is in
doSnapshotofmonitorStreamandmonitorConsumer(andmonitorCluster), so we must not attempt installing snapshots outside of those.Signed-off-by: Maurice van Veen github@mauricevanveen.com