Support ephemeral instances

[Nothing4You]

Hi,

I'm not sure how reasonable this is to implement, but it would be great if this could be used with instances that can only run a single pipeline before being destroyed.

Implementation wise, this may be doable by deploying a new instance, waiting until it has accepted its first job, then disabling the agent in woodpecker, which will prevent it from picking up new jobs.

This would allow running CI jobs automatically even in less trusted environments, as every pipeline will have its own clean environment.

[windsource]

Hi @Nothing4You,

Picus current supports AWS and Hetzner. The AWS agent provider is using one instance which is started when new jobs are pending and stopped when all jobs have been done. So the same instance is used all over. In order to support ephemeral instances the concept for the AWS agent provider in Picus would need to be changed.

The Hetzner cloud agent provider in Picus starts a new instance when a job is pending so here we have a clean start situation. The instance is shutdown when there no more jobs for a certain time. For Hetzner cloud you pay for every hour that the server is running so when you shut it down after a few minutes you still have to pay for a whole hour AFAIK.

So having ephemeral instances for a less trusted envionment might be possible in general but has some implications.

[Nothing4You]

I see, I'm primarily a Hetzner Cloud user, so I didn't realize that it's handled differently for AWS. It is also my understanding that you have to pay a full hour for each started HC instance, so there would indeed be increased costs depending on workloads if this setting was enabled.

As a reference, GitLab CI supports setting MaxBuilds when using docker-machine based autoscaling: https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-runnersmachine-section

Maximum job (build) count before machine is removed.

[SeanLatimer]

For Hetzner cloud you pay for every hour that the server is running so when you shut it down after a few minutes you still have to pay for a whole hour AFAIK.

Hetzner bill for each machine whether it's running or not, as the resources are reserved for you.

Ideally, the scaler should check agent runtime and only delete an idle agent near the top of the current billing hour to minimize costs.

[Nothing4You]

Ideally, the scaler should check agent runtime and only delete an idle agent near the top of the current billing hour to minimize costs.

That seems like a separate issue for optimizing hetzner cloud costs.

This issue is explicitly for discarding VMs after every single build job, at the cost of paying for the full hour regardless, to allow untrusted builds to be performed in clean environments, without risking them being able to impact any other builds.