search

buildkite / elastic-ci-stack-for-aws

An auto-scaling cluster of build agents running in your own AWS VPC
https://buildkite.com/docs/quickstart/elastic-ci-stack-aws
MIT License
414 stars 265 forks source link

Documentation clarification: what happens if no spot instances are available? #1296

Open JacobEvelyn opened 4 months ago

JacobEvelyn commented 4 months ago

Describe the bug In this documentation, I see this under OnDemandPercentage:

Default is 100% OnDemand - reduce this to use some Spot Instances when they're available and cheaper than the OnDemand price.

This implies that if no spot instances are available, we'll fall back to on-demand—even with OnDemandPercentage configured to 0.

However, I also see this issue, which says that if no spot instances are available it will continue trying indefinitely. However, that issue is several years old and I'm not sure if it's still accurate.

Which is correct?

Expected behavior Clearer documentation about what happens when no spot instances are available and the OnDemandPercentage has already been met (or is 0).

To be clear, my desired behavior is that it falls back to on-demand and doesn't continue trying spot indefinitely.

moskyb commented 4 months ago

g'day @JacobEvelyn! hopefully i can try to clear this up a little:

first off, at time of writing the elastic stack will never fall back from spot to ondemand - this is a tricky thing to do in EC2, as AWS really doesn't want to cause capacity crunches. it is technically feasible to do it, but it would require a lot of custom scaling logic.

to clarify the docs a little, the way this setting works is that the percentage of spot instances is best-effort - ie, dependent on pricing/availability etc - and the percentage of ondemand instances is guaranteed.

as an example, if i had an elastic stack with a desired count of 10 and an OnDemandPercentage of 70, if there are spot instances available, i'll have 7 on-demand instances and 3 spot instances in the elastic stack ASG. If no spot instances are available, we'll just have the 7 on-demand instances and no spot instances.

i hope this answers your question - let me know if you have any followups.

JacobEvelyn commented 4 months ago

Thanks @moskyb! If there's a way to clarify that in the docs that would be helpful.