search

hetznercloud / hcloud-go

A Go library for the Hetzner Cloud API
https://pkg.go.dev/github.com/hetznercloud/hcloud-go/v2/hcloud
MIT License
365 stars 45 forks source link

Hitting the Hetzner API limits #79

Closed gliwka closed 6 years ago

gliwka commented 6 years ago

Hi,

I'm currently using the terraform provider to deploy my infrastructure. Due to the nature of my project, sometimes I need to start a cluster (26 nodes), delete it, then start it again within the same hour.

During this I always hit the 3600 requests / hour API limit enforced by Hetzner in the middle of starting the new cluster after shutting down the old one.

This would mean the terraform provider somehow issues (3600 Calls / 3x (Start, Stop, Start) / 26 Nodes) => more than 46 calls per node to hit this limit assuming an even distribution.

While investigating this client library the terraform provider is based upon, the following section grabbed my attention: https://github.com/hetznercloud/hcloud-go/blob/e37d00121bda0fdf86342d47b9ac48cbd79e2c27/hcloud/action.go#L151

It seems that every 500ms a call is being made to check if an action has been completed. It furthermore seems, that those calls count against the API limit, too. This would explain the rate limit issue.

If this is the case, would it be possible to expose the "magic number" 500ms as a configuration option?

Kind regards, Matthias Gliwka

thcyron commented 6 years ago

We have an idea how to solve this problem the right way (less polling).

If that’s a blocking issue for you right now, I guess you have to build the Terraform plugin on your own with a greater polling interval.

gliwka commented 6 years ago

That's even better :-)

I've already compiled my own, so it's not a blocker for me.

thcyron commented 6 years ago

The action endpoint now uses its own ratelimit pool. While you may still run into ratelimit errors when polling an action’s status/progress, it no longer affects other activities like creating servers.