search

kumahq / kuma

🐻 The multi-zone service mesh for containers, Kubernetes and VMs. Built with Envoy. CNCF Sandbox Project.
https://kuma.io/install
Apache License 2.0
3.61k stars 332 forks source link

Be less aggressive on some reconciliation failure #9189

Open lahabana opened 7 months ago

lahabana commented 7 months ago

What happened?

Whenever you hit an issue in the reconcilation step (2024-02-09T13:36:50.755Z ERROR xds-server.dataplane-sync-watchdog OnTick() failed {"dataplaneKey": {"Mesh":"default","Name":"edge-gateway-79b4654bcf-5tx6t.kuma-demo"}, "error": "could not reconcile: failed to generate a snapshot: imports[0]{name=\"gateway-proxy\"}: generator.generator failed: ...) of a dataplane we retry in a loop very aggressively (from https://github.com/kumahq/kuma/issues/9188).

This causes the CP to hog the CPU and also will start failing healthchecks and restart. The handling should more graceful to avoid this.

jakubdyszkiewicz commented 7 months ago

Triage: to repro, do steps from #9188

github-actions[bot] commented 4 months ago

This issue was inactive for 90 days. It will be reviewed in the next triage meeting and might be closed. If you think this issue is still relevant, please comment on it or attend the next triage meeting.

github-actions[bot] commented 1 month ago

This issue was inactive for 90 days. It will be reviewed in the next triage meeting and might be closed. If you think this issue is still relevant, please comment on it or attend the next triage meeting.