After a large number of watch connections are disconnected from a client at the same time, the new watch cannot work properly.

alterge1st commented 1 week ago

Bug report criteria

[ ] This bug report is not security related, security issues should be disclosed privately via etcd maintainers.
[ ] This is not a support request or question, support requests or questions should be raised in the etcd discussion forums.
[ ] You have read the etcd bug reporting guidelines.
[ ] Existing open issues along with etcd frequently asked questions have been checked and this is not a duplicate.

What happened?

We used the in-process etcdserver of v3client. Then we created a client, created a watch connection to the same resource every second, without freeing them, and ran it for more than 1 minute. When the client maintains a large number of watch connections, we kill the client process. After the client process is killed, when other clients attempt to establish watch connections for the same resource, the new watch connections cannot obtain new event changes.

What did you expect to happen?

After the client is killed, the new watch connection for the same resource can properly listen to event changes. And after analysis, the blocking problem exists. Although it is unreasonable for the client to establish a large number of watch connections with the same resource at the same time, can the etcd server do something to avoid the blocking?

How can we reproduce it (as minimally and precisely as possible)?

We created a large number of Watch connections to the same configmap resource in a loop through a process using code similar to the following: main.txt After running this program for 1 minute, kill the program. When you continue to run the kubectl get configmap -A -w command, after the configmap is modified, the configmap change cannot be watched.

Anything else we need to know?

After the client is killed, a large number of watch connections are disconnected. The code analysis shows that the Send() function of WatchCancelRequestin case ws := <-w.closingc of the (w *watchGrpcStream) run() method in etcd/client/v3/watch.go is blocked and unable to continue processing. It is suspected that a large number of WatchCancelRequests cause the channel in watchGrpcStream to be fully occupied. As a result, new WatchResponse cannot be pushed into sws.ctrlStream. The WatchResponse obtained from ctrlStream and new WatchResponseare blocked in case pbresp := <-w.respc and case ws := <-w.closingc in (w *watchGrpcStream) run().

Etcd version (please run commands below)

```console $ etcd --version # 3.5.11 $ etcdctl version # 3.5.11 ```

Etcd configuration (command line flags or environment variables)

# paste your configuration here

Etcd debug information (please run commands below, feel free to obfuscate the IP address or FQDN in the output)

```console $ etcdctl member list -w table # paste output here $ etcdctl --endpoints= endpoint status -w table # paste output here ```

Relevant log output

No response

alterge1st commented 1 week ago

When the size of channel ctrlStream(ctrlStreamBufLen) is increased, we need to create more watch connections and disconnect them to reproduce the blocking problem.

ahrtr commented 1 week ago

We recently fixed a watch related goroutine leak issue, https://github.com/etcd-io/etcd/pull/18784

The fix will be included in 3.5.17, which is supposed to be released this week. Please try again with the new version once available.

alterge1st commented 1 week ago

We recently fixed a watch related goroutine leak issue, #18784

The fix will be included in 3.5.17, which is supposed to be released this week. Please try again with the new version once available.

Okay, thanks, we'll try the new version

alterge1st commented 1 week ago

We recently fixed a watch related goroutine leak issue, #18784

The fix will be included in 3.5.17, which is supposed to be released this week. Please try again with the new version once available.

Unfortunately, I modified my local code to follow the latest changes, but the problem persists.

serathius commented 1 week ago

Not following whether the issue is etcd or K8s related. In the repro you provide and discuss code for kubernetes API while the folowing debugging is about etcd. I would like to clarify this, because K8s apiserver demultiplexes watch connections to etcd. So the issue with client cancelation should not happen for K8s, as 100 watches opened to apiserver still opens only 1 watch to etcd.

alterge1st commented 1 week ago

Not following whether the issue is etcd or K8s related. In the repro you provide and discuss code for kubernetes API while the folowing debugging is about etcd. I would like to clarify this, because K8s apiserver demultiplexes watch connections to etcd. So the issue with client cancelation should not happen for K8s, as 100 watches opened to apiserver still opens only 1 watch to etcd.

Kube-apiserver is integrated with etcd. The in-process server of etcd is used to directly invoke APIs instead of using etcd service ports. This problem occurs when the watch command of K8s is used. When another process is started to create a large number of watch requests for the configmap cyclically, killing the process will cause the watch command of the configmap by the Kubernetes to become invalid.

serathius commented 1 week ago

The in-process server of etcd is used to directly invoke APIs instead of using etcd service ports.

This should not change the fact that apiserver will demultiplexes watch, or have you disabled watch cache?

alterge1st commented 1 week ago

The in-process server of etcd is used to directly invoke APIs instead of using etcd service ports.

This should not change the fact that apiserver will demultiplexes watch, or have you disabled watch cache?

Yes, we did disable the watch cache.

alterge1st commented 3 days ago

If ctrlStreamBufLen is set to hundreds, will etcd run properly?

ahrtr commented 2 days ago

Could anyone create a test using etcd client SDK instead of k8s client-go to reproduce this?

etcd-io / etcd