search

kumahq / kuma

🐻 The multi-zone service mesh for containers, Kubernetes and VMs. Built with Envoy. CNCF Sandbox Project.
https://kuma.io/install
Apache License 2.0
3.64k stars 331 forks source link

Improve CP components shutdown sequence #8500

Open slonka opened 10 months ago

slonka commented 10 months ago

Description

Similar to https://github.com/kumahq/kuma/issues/1001 we experience issues while shutting down the CP. Connections are broken, we are unable to flush things. I think in the past we talked about having some sort of hierarchy / dependency tree for components so we can shut down things in proper order

image

github-actions[bot] commented 7 months ago

This issue was inactive for 90 days. It will be reviewed in the next triage meeting and might be closed. If you think this issue is still relevant, please comment on it or attend the next triage meeting.

jakubdyszkiewicz commented 5 months ago

Seems to not be a problem now.

slonka commented 3 months ago

The only thing that still pops up during restarts rarely is:

FAILED TO ACK A DISCOVERY RESPONSE: RPC: KDSSyncClient finished with an error: failed to ACK a discovery response: rpc error: code = Unavailable desc = transport is closing
github-actions[bot] commented 3 months ago

Removing closed state labels due to the issue being reopened.

slonka commented 3 months ago

We shouldn't log 

error: unavailable: cannot run query to read record version number: failed to connect to `user=x database=x`: y:5432 (z): dial error: dial tcp y:5432: operation was canceled

during shutdown

github-actions[bot] commented 3 weeks ago

This issue was inactive for 90 days. It will be reviewed in the next triage meeting and might be closed. If you think this issue is still relevant, please comment on it or attend the next triage meeting.