Add support for AWS spot fleets

buildkite / elastic-ci-stack-for-aws

An auto-scaling cluster of build agents running in your own AWS VPC

https://buildkite.com/docs/quickstart/elastic-ci-stack-aws

MIT License

417 stars 267 forks source link

Add support for AWS spot fleets #147

Closed shuber closed 3 years ago

shuber commented 7 years ago

http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/spot-fleet.html

We ran into issues today when the spot price for the instance type that we're using skyrocketed. We ended up just spinning up a separate stack with an alternative instance type in the meantime. It would be cool to be able to specify a few different instance types and let spot fleet handle issues like these.

screenshot 2016-09-19 11 42 26

lox commented 7 years ago

I'd suggest running two stacks, one spot and one non-spot. The issue with spot fleet is that (as of last time I looked) there wasn't a way to scale it based on need.

toolmantim commented 7 years ago

@lox I think that changed somewhat recently: https://aws.amazon.com/blogs/aws/new-auto-scaling-for-ec2-spot-fleets/

lox commented 7 years ago

Oh I totally forgot about that, apologies. Happened whilst I was on holidays :)

In that case, I'm on it.

lox commented 7 years ago

Ok, am starting to dev on this now. Of course the complexity is that each region has slightly different spot instances available.

caarlos0 commented 7 years ago

Can I help with anything?

atyndall commented 7 years ago

+1. This would be great for us also, being able to choose several instance types around the same performance threshold will help us avoid spot price fluctuations.

lox commented 7 years ago

Starting work on this in ##269

tspacek commented 6 years ago

For anyone who wants to give this a shot manually (outside of Cloudformation): make a new Spot Fleet request, copy the config from Buildkite's ASG Launch Configuration, then hook the Spot Fleet into the same auto-scaling events. Turn off the ASG. Done.

Only wrinkle so far is that the number of agents per instance isn't aware of the number of vCPUs, as far as I know.

jtoy commented 4 years ago

just checking in on this :)

KevinGrandon commented 4 years ago

Update: After being unable to utilize the steps above, I believe we have a temporary way to scale across multiple instance types using these following steps.

1) Create a Launch Template.

Navigate to EC2 -> Launch Configurations
Select the Launch Template created by the elastic stack
Click Copy to launch template, click Sumbit.

2) Update Launch Template to not use spot Instances

Navigate to EC2 -> Launch Templates
Select the created Launch Template and click edit (create new version)
Deselect Request Spot Instances and save.

3) Update the Autoscaling Group

Navigate to EC2 -> Autoscaling Groups.
Click on the Autoscaling Group created by the Elastic Stack.
Click edit.
Change Launch Instances Using to Launch Template.
Template: Select the launch templated we created in Step 1.
Fleet composition: Combine purchase options and instances.
Instance types: Select all desired instances.
On-Demand Percentage: specify desired amount of on-demand instances (we use 0%)
Save

keithduncan commented 3 years ago

With the multi-instance type functionality exposed through Launch Templates and Auto Scale groups allowing instance requests to span on-demand and spot, I don’t think there’s a need to integrate EC2 Fleets directly anymore 🥳