nats-io / nats.rs

Rust client for NATS, the cloud native messaging system.
Apache License 2.0
1.07k stars 169 forks source link

Fix KV create race after delete/purge #1301

Closed fnichol closed 3 months ago

fnichol commented 3 months ago

This change fixes an issue when using a JetStream K/V store where a user is creating, deleting, and re-creating keys. If the last entry for a key is a Operation::Delete or Operation::Purge, the initial self.update() returns an error, causing the second part of the method to be exercised.

Prior to this change, if the entry was deleted or purged a kv.put() call is used which ignores the revision of that last entry. A single writer to the K/V store would succeed (as no other writers would write first) so no problem. However, if 2 writers attempt to create a key, then a second writer could call the kv.put() before the first writer calls kv.put(). This means that both writers get an Ok(revision) and can assume that they won the creation of the key.

When using a "distributed lock" pattern (that is many writers race to create a key and the first successful writer wins), this above scenario results in potentially more than one writer who believes they have uniquely acquired the distributed lock.

This change replaces the kv.put() call to a kv.update() call and provides the revision from the deleted/purged entry to ensure that no other writer has beaten the caller to this update. This change closes the race period between concurrent writers to between the first update and the second update call with some optimistic write concurrency to detect another writer.

It appears as though this strategy is in effect in the Go client code kv.Create implementation.