search

testflows / TestFlows-GitHub-Hetzner-Runners

Autoscaling Self-Hosted GitHub Actions Runners on Hetzner Cloud.
https://testflows.com
Other
37 stars 3 forks source link

Removing recycle server instances that use dedicated vCPU #5

Closed MyroTk closed 6 months ago

MyroTk commented 1 year ago

Recycled servers count against the Heztner D.vCPU limit and prevent new servers from being created.

For example, if the the D.vCPU limit is 32 and a job that uses ccx53 follows a job that uses ccx43, the ccx43 instance is going to sit in a powered off state waiting to be recycling, preventing the ccx53 from being created.

vzakaznikov commented 1 year ago

Yes, this is a problem. Right now the workaround is just to use the same dedicated server types or turn off server recycling.

Not sure how to exactly fix it. Need to look into this. Right now we don't have a logic to check which server uses which resources. Maybe we could start killing random recyclable servers if we see an API exception that implies that we've hit some limit.

vzakaznikov commented 1 year ago

Should be partially covered by the https://github.com/testflows/TestFlows-GitHub-Hetzner-Runners/releases/tag/1.5.231010.1002212.

However, right now we don't check if any of the recyclable server have ccx type. That's something that still needs to be added. We will just either remove a recyclable server at random or the cheapest one.

vzakaznikov commented 6 months ago

Not planned for now.