search

hetznercloud / hcloud-cloud-controller-manager

Kubernetes cloud-controller-manager for Hetzner Cloud
Apache License 2.0
740 stars 118 forks source link

fix: Too many requests to `GET /v1/servers/{id}` #661

Open apricote opened 5 months ago

apricote commented 5 months ago

TL;DR

In some situations hcloud-cloud-controller-manager starts to spam GET /v1/servers/{id} for a subset of the nodes in the cluster every few seconds.

Expected behavior

I would expect hccm to only send a single request per --node-status-update-frequency (defaults to 5 minutes).

Observed behavior

We get reports from customers that exhaust the rate limit and we can see that hcloud-cloud-controller-manager sends > 1 req/s for GET /v1/servers/{id}. The default rate limit is 1 request per second.

A restart of the pod fixes the behaviour.

Minimal working example

We are not sure how to reproduce this yet.

Log output

No response

Additional information

Looking at some request logs, this seems to affect even very old versions of HCCM (>2 years).

github-actions[bot] commented 2 months ago

This issue has been marked as stale because it has not had recent activity. The bot will close the issue if no further action occurs.