search

gardener / etcd-backup-restore

Collection of components to backup and restore the etcd of a Kubernetes cluster.
Apache License 2.0
286 stars 100 forks source link

Delay in updating peer TLS enabled status can result in more than desired restarts of etcd #538

Open unmarshall opened 2 years ago

unmarshall commented 2 years ago

Describe the bug: Background:

Refer: https://github.com/gardener/etcd-backup-restore/issues/529 For most cases this works. To determine how many restarts of etcd member is required depends on the timely updation of member lease annotation. In an edge scenario, it is possible that etcd-druid does not see the update of the lease in time and schedules additional restarts. This can result in a total of 4 restarts for a etcd cluster (single member) instead of desired 2. We saw this issue once and currently its hard to reproduce this issue.

Since there are two actors required currently in the process of ensuring that peer TLS enablement is reflected in the running etcd process we would like to evaluate if we can remove one actor completely by using cooperating containers in the etcd pod - > https://kubernetes.io/docs/tasks/configure-pod-container/share-process-namespace/

Additional restarts of statefulsets in general are risky as it involves detach<->attach of PVs and we have seen issues with this on different providers. So number of times we recreate statefulset should be reduced to the bare minimum.

Expected behavior: There should be a max 1 restart triggered by etcd-druid and the second restart should be done within the etcd via the etcd-backup-restore container.

How To Reproduce (as minimally and precisely as possible):

Logs:

Screenshots (if applicable):

Environment (please complete the following information):

Anything else we need to know?:

unmarshall commented 2 years ago

/assign

shreyas-s-rao commented 1 year ago

/assign @seshachalam-yv

ishan16696 commented 1 year ago

While writing a doc for current scale-up design in this PR: https://github.com/gardener/etcd-druid/pull/606 I went through the etcd provided doc: https://etcd.io/docs/v3.4/op-guide/runtime-configuration/#update-advertise-peer-urls It says:

To update the advertise peer URLs of a member, first update it explicitly via member command and then restart the member. The additional action is required since updating peer URLs changes the cluster wide configuration and can affect the health of the etcd cluster.

IMO, they are suggesting to restart the member as updating peer URL(s) is a cluster wide operation but in our case cluster size is 1 not greater than 1 which will get update by API call and when we update the volume mount etcd-0 for peer URL certificate, then pod will anyway get restart, so in my opinion we don't need to do 2nd restart of etcd pod.