sritchie
commented
12 years ago Yeah, might be good to figure out how much space we need for our jobs and boot up core nodes that cover that size space on HDFS (taking into account the 3x replication factor).
I've ALWAYS run spot-only clusters w/ my twitter work, and we've almost never had a cluster shut down. I'd say spot-only is worth the huge price savings.
On Mon, Jul 23, 2012 at 2:26 PM, dpetrovics < reply@reply.github.com
wrote:
Hey Guys I cleaned up the forma-deploy project a bit. You can now do: lein run --emr --type large --size 5 --bid 0.05 to use a bid price of 5 cents for the core nodes. (The master node always uses on-demand). If you do NOT supply a --bid, it will use the on-demand price for the core nodes. Please not that the bid price is a MAXIMUM price that we're willing to pay, what we actually pay is the current SPOT price, which Amazon sets. As long as our bid > spot we are OK. If the spot price goes above our bid, our instances using the spot price are terminated.
I also cleaned up some of the validation and the way we parse the command line arguments so that all of the error messages are sensible.
Question-- on Amazon it suggests that we use on-demand for master and core, keep these to a minimum, and start up a bunch of task nodes and use the spot price for these. If they get terminated then we're still OK since their tasks can be reassigned. I would recommend that we either do this, or keep our bid prices really high so our core nodes are not getting terminated. Thoughts? Which option?
Also, since we have only been using EMR, we're not using about 90% of the code in here that was written for EC2 and pallet-hadoop. The EMR stuff just requires us to call the elastic-mapreduce script with the proper arguments. Just in case, I updated the code so we can specify bid prices for the --start option as well, which does not use EMR.
Not sure if we should just keep all the extra code, spin it off to a new repo, or what, but it would be good to discuss what features you guys want have available in this project. There's also a few leiningen plugins that Sam wrote (to upload the project uberjar, for example), but they're not being used since we're using Robin's bootstrap script from S3. What to do with these?
Thanks!
You can merge this Pull Request by running:
git pull https://github.com/dpetrovics/forma-deploy master
Or you can view, comment on it, or merge it online at:
https://github.com/reddmetrics/forma-deploy/pull/3
-- Commit Summary --
- Changed the bootstrap file to forma_bootstrap_robin.sh. I think this is the updated one that clone our git repository into the cluster. Tweaked the options map in parse-hadoop-args. The parse function was throwing an unhelpful error in the clojure.tools.cli library in every situation when we dont provide a --size (even if we just want to stop a cluster). Now it provides sensible error messages using our three validators: just-one? size-valid? and spotprice-valid? Added an option --spot which you can use to specify a spot price.
- changed spot price to bid price, and added this parameter to all necessary functions as an optional argument (or required, as appropriate)
- small tweaks, and accounted for a nil bid price
- support for on-demand pricing
-- File Changes --
M src/forma/hadoop/cli.clj (3) M src/forma/hadoop/cluster.clj (92)
-- Patch Links --
https://github.com/reddmetrics/forma-deploy/pull/3.patch https://github.com/reddmetrics/forma-deploy/pull/3.diff
Reply to this email directly or view it on GitHub: https://github.com/reddmetrics/forma-deploy/pull/3
Sam Ritchie, Twitter Inc 703.662.1337 @sritchie09
(Too brief? Here's why! http://emailcharter.org)
dpetrovics
danhammer
robinkraft
eightysteele
Hey Guys I cleaned up the forma-deploy project a bit. You can now do: lein run --emr --type large --size 5 --bid 0.05 to use a bid price of 5 cents for the core nodes. (The master node always uses on-demand). If you do NOT supply a --bid, it will use the on-demand price for the core nodes. Please not that the bid price is a MAXIMUM price that we're willing to pay, what we actually pay is the current SPOT price, which Amazon sets. As long as our bid > spot we are OK. If the spot price goes above our bid, our instances using the spot price are terminated.
I also cleaned up some of the validation and the way we parse the command line arguments so that all of the error messages are sensible.
Question-- on Amazon it suggests that we use on-demand for master and core, keep these to a minimum, and start up a bunch of task nodes and use the spot price for these. If they get terminated then we're still OK since their tasks can be reassigned. I would recommend that we either do this, or keep our bid prices really high so our core nodes are not getting terminated. Thoughts? Which option?
Also, since we have only been using EMR, we're not using about 90% of the code in here that was written for EC2 and pallet-hadoop. The EMR stuff just requires us to call the elastic-mapreduce script with the proper arguments. Just in case, I updated the code so we can specify bid prices for the --start option as well, which does not use EMR.
Not sure if we should just keep all the extra code, spin it off to a new repo, or what, but it would be good to discuss what features you guys want have available in this project. There's also a few leiningen plugins that Sam wrote (to upload the project uberjar, for example), but they're not being used since we're using Robin's bootstrap script from S3. What to do with these?
Thanks!