brandond
commented
12 months ago With the native etcd implementation in clustered mode, are updates serialized per key? Reads/watches are always going to be stale when a write is attempted, so it seems like if the loop is to attempt, then refetch, then retry, this would exacerbate the problem.
Kubernetes uses an etcd transaction to ensure that the key revision is the same as what was expected. If the revision matches, then the key is written to. If it does not match, then the current value is returned, and the client gets a conflict error along with a copy of the current object state. The client is expected to merge their intended changes into the current object state, and retry.
If NATS is not honoring this transaction behavior, then there is probably a significant potential for incorrect behavior.
You can find the client's transaction at https://github.com/kubernetes/kubernetes/blob/3cf3702d1e638548ba9fcfd957ebae43e1a573cb/staging/src/k8s.io/apiserver/pkg/storage/etcd3/store.go#L527-L533
bruth
sttts
dereknola
@brandond Quick question on desired
Updatesemantics. The previous implementation of the NATS backend largely assumed a single node since it held a lock during the Update call. It did a read from the NATS KV, did some checks, and then applied a KV Put, which performs a blind write.However, in a clustered setup, e.g. three k3s servers, each will be watching, competing for leases, etc. so interleaved writes can definitely be attempted. NATS KV has a native Update operation that takes the "expected last sequence/version" of that key to effectively do a conditional write, to prevent last-writer-wins behavior, i.e. serializability per key.
All that said, while watching the logs for which sets of keys get aggressively updated, it seems like it can be problematic, such as
/registry/apiregistration.k8s.io/apiservices/v1beta1.metrics.k8s.io.With the native etcd implementation in clustered mode, are updates serialized per key? Reads/watches are always going to be stale when a write is attempted, so it seems like if the loop is to attempt, then refetch, then retry, this would exacerbate the problem.