Closed Fluxx closed 13 years ago
Looks great. I pulled it in to the main repo. Could you add an example to examples/ as well?
re: error handling, I haven't used spot instances but off hand I'd say it's probably better without the extra functionality.
Thanks! I'll take a stab at the writing an example. I ran the test suite on Friday and got it to run, though not all the tests passed and it surprised me that it nuked all the keypairs in my AWS account :/ I'll see if I can get a few tests written though.
Also, on the test suite, not sure if you knew or were interested, but there is this great gem called VCR that can record HTTP interactions for tests and then replay them from saved files. That helps keep fast and from needing to talk to the EC2 API and actually change state there during its run. Thought it might help here.
Ohhhh I just realized you meant examples of configuration as opposed to tests. I'll add that to another pull request :)
Crap, sorry about the keypairs. I built the tests to run using non-production credentials. I disabled that specific test and added a message to make that more clear.
Thanks for mentioned VCR. I'll check it out.
I tested it out. A couple things we'll need to take care of before releasing it:
launched_availability_zone
.m1.small
.When you cancel the spot request, it needs to send a CancelSpotInstanceRequestsResponseSetItemType request. Otherwise, the requests will continue and eventually be fulfilled but Rudy won't know about it (and it'll silently charge you in the background...)
Will do.
All instances are starting up in us-east-1a, regardless of the config. I think you also need to send the parameter launched_availability_zone. The instance type is always m1.small.
Hrm, that's weird. I already started some spot-based instances in our project is us-west and with other sizes (m1.large, m2.xlarge, m2.2xlarge, etc) and they started fine. I'll try reproducing with the most latest code and see what's up.
So I'm trying to repro the but where region/instance size is not respected from the config, and am unable to do so. I've tried a few combinations of globals
vs machines
config and it seems to be picking up all my changes. Can you see if you're still affected by it, and if so, send me your exact Rudyfile, revsion sha and repro steps (commands, etc) so I can see if I can repro it?
P.S. I'm gonna work on the CancelSpotInstanceRequestsResponseSetItemType fix right now, so expect a pull request for that soon.
I haven't forgotten about this btw. I'm just in the midst of moving blamestella.com off of EC2. Should be able to follow up again later this week.
Hey, any updates? I have more time now so let me know if you want some help with it.
I think things are is pretty good shape. We've been using our own fork internally for a couple weeks now and things seem to be happy. There is one bug with the spot requests. I'm making the spot requests with a :availability_zone_group
tag to the EC2 API, which just tags all the machines in the group with a specific tag to guarantee that they all boot in the same AZ, not in a specific AZ. I'll need to change that to use the right option to actually use the specified AZ.
Here is the beginnings of the integration of spot instance requests. I'm not done with everything, I still have to add some command line options and error checking, but the meat of the functionality is there. Hope to use this pull request to get feedback and manage development of this feature before merging it back.
To enable spot instances, I create a
pricing
config option for yourRudyfile
. Valid options are:on_demand
(for normal instances) and:spot
for spot instances. If you use:spot
, you need to pass a block where you set the your bid price, i.e:The additions to utilize spot instances are as follows:
startup
routine, it checks to see if spot instances are needed.Rudy::Routines::Handlers::SpotRequest.create
to kick off the call to the EC2 API to create the spot instance request, and then wait for the spot requests to be fulfilled (instance booted).SpotRequest.create
method returns an array ofRudy::AWS::EC2::SpotRequest
objects, which are passed toRudy::Machines.from_spot_request
, which actually makes and saves the machines to SDB.RyeTools.create_set
(like they would be fromRudy::Machines.create
), and thestartup
routine goes on its merry way.If the spot requests are never fulfilled, the user can abort the startup routine and then do whatever they like. I had thoughts about adding functionality to cancel the spot request and/or changing to on demand instances as backup, but hadn't done that yet. That's part of the "error handling" I haven't finished yet. Eager to hear what you guys think?
Also, to make it easier for me to work on the project I added a Gemfile to the project and change the shebang on the bin files to
#!/usr/bin/env ruby
works more universally. The previous shebang of#!/usr/bin/ruby
didn't work for me (and won't for lots of other people) cause I use RVM. Happy to rebase my branch and remove the gemfile or shebang commits however, if you'd like.