aerospike / aerospike-client-go

Aerospike Client Go
Apache License 2.0
429 stars 199 forks source link

Is Key.digest unique server hash for different namespaces? #363

Closed xqzhang2015 closed 2 years ago

xqzhang2015 commented 2 years ago
type Key struct {
    // Unique server hash value generated from set name and user key.
    digest [20]byte
}
  1. Is Key.digest unique in a namespace?
  2. Is Key.digest unique for different namespaces? It seems Key.computeDigest() doesn't care about Key.namespace.

The requirement is to add singleflight Get() for the concurrent access to the same key. I plan to wrap the Get() func and use Key.digest as the record ID. Hotkey is easily happening sometimes in our system.

khaf commented 2 years ago

Digests are unique in the namespace (within reason, understanding what a hash represents). When you try to retrieve a key, the command sends the namespace as part of the request to the server, so it does not need to be a part of the digest. Hot keys are the result of trying to read|write the same key concurrently by a lot of actors. It is orthogonal to the hash uniqueness problem.

xqzhang2015 commented 2 years ago

@khaf thanks for the reply.

To ease the hotkey from high reading/writing concurrency, we could set one singleflight.Group for each namespace and use the Key.digest as the ID for one reading/writing op, right?

https://github.com/golang/go/blob/master/src/internal/singleflight/singleflight.go#L47

Or do you have any proposal for easing hotkey? BTW, is it good to add such a reading singleflight attribute to the aerospike package, as configurable Policy?

khaf commented 2 years ago

Singleflight could in theory ease the pressure off hot keys. Keep in mind that singleflight itself uses a hash map internally with a lot of additional wiring including channels, which depending on your app could add significant overhead by itself. You need to benchmark your use case to make sure the overhead does not interfere with your QoS.

xqzhang2015 commented 2 years ago

@khaf thanks for the comment. I will try it with benchmark.