search

nchammas / flintrock

A command-line tool for launching Apache Spark clusters.
Apache License 2.0
638 stars 116 forks source link

Slave-only spot instances #82

Open nchammas opened 8 years ago

nchammas commented 8 years ago

Flintrock supports launching clusters using spot instances. However, when doing so, all the cluster instances are launched as spot instances.

Generally, though, you don't want the master to be a spot instance. You want it to stick around even if various slaves are dropped so that your job can still chug along (though at a slower pace) and so that you can re-add slaves to the cluster at a later time.

We should change our support for launching clusters on spot instances such that only the slaves are spot instances.

This work (kinda) depends on #16.

engrean commented 8 years ago

@nchammas would this solve the problem that occurs when launching a spot instance cluster and one of the nodes ends up getting terminated which ends in the launch command just hanging forever? Or should I submit a new issue for that case?

nchammas commented 8 years ago

@engrean - Have you seen this issue since d5b086c81952aaf78dd7febfedc6b424e63e3697 made it in? Flintrock should now immediately error out as soon as a spot request fails.

You'll need to be running on master to get this fix.

If you're still seeing the issue, the yes, please submit a new issue so we can look into it. I'd be curious to see where exactly the launch hangs, for example.

sylvinus commented 8 years ago

Would love to have this feature. It would be great to be able to specify a different instance type for the master as well.

ktdrv commented 7 years ago

Would it make sense to separate the add-slaves logic from launch? In other words, the workflow would be to create a master instance only first and then add all the slaves of any type you want to it with a second call.

DBCerigo commented 7 years ago

Seconding the "would love this feature", and even more so, being able to specify different master and slave instance types.

mblackgeo commented 5 years ago

@nchammas is there any progress on this?

nchammas commented 5 years ago

@mblack20 - Nope. The current workaround, I believe, is to launch a regular cluster with no slaves, and then add spot slaves separately using add-slaves.