search

etcd-io / etcd

Distributed reliable key-value store for the most critical data of a distributed system
https://etcd.io
Apache License 2.0
47.85k stars 9.77k forks source link

Using multiple endpoints vs relying on k8s load balance #13097

Closed elbaro closed 3 years ago

elbaro commented 3 years ago

We have a set of etcd nodes (etcd-1, etcd-2, ..) in our k8s cluster. Accessing the hostname "etcd" (k8s deployment) connects to one of them.

Unfortunately a popular python client (https://github.com/kragniz/python-etcd3) does not support multiple endpoints, and we rely on k8s's load balancing. I wonder this approach is inferior to the native multiple endpoint support by the etcd client.

For example, if one etcd responds with error without network disconnection, the k8s load balance still thinks the etcd healthy and may keep forwarding the request to this problematic node. Or, in a case of network partition, k8s may keep forwarding the traffic to a node in the minor parition without leader. Can multiple endpoints detect this and use a better endpoint?

  1. Does giving multiple endpoints(etcd-0,etcd-1,..) to etcd clients work better than relying on k8s service (etcd) ?
  2. If I provide 1 endpoint to the etcd-client, does it automatically populate the endpoint list with the cluster members?
stale[bot] commented 3 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 21 days if no further activity occurs. Thank you for your contributions.