search

nchammas / flintrock

A command-line tool for launching Apache Spark clusters.
Apache License 2.0
636 stars 116 forks source link

Hybrid Spot / Non Spot instances #233

Closed damageboy closed 6 years ago

damageboy commented 6 years ago

Hi, I'm opening this issue mostly to ask my way around as the gitter seems pretty dead to me.

I'm interested in eventually submitting a PR that would allow to launch at the very least the master node, and perhaps a few additional nodes (though mainly the master) as a non spot-instance while allowing the slave nodes to be launched as spot instances.

This looks to me, to be a relatively very easy change in the code. In fact, it looks so simple, that I feel I must be missing something as to why it wasn't just done this way in the first place...

Could any one knowledgeable explain to me where I'm missing something trivial?

Is this a feature that would interest users of flintrock?

nchammas commented 6 years ago

Hi @damageboy! Yes, this would be a good change to make to Flintrock. In fact, I think this issue is a duplicate of #82.

I would say the main reason it hasn't been done already is that no-one has stepped up to do it! If you would like to contribute an improvement here, I would be happy to guide you through the process. To keep things simple, I would make only the master launch as an on-demand instance when a spot price is set.

damageboy commented 6 years ago

I'll get to work on it next week. Sorry if this is a stupid question, but is there an aws simulator service, or is everything done on production?

nchammas commented 6 years ago

Not a stupid question at all. I think you are looking for a library to mock out the AWS API. There are a few (like moto), but Flintrock doesn't use them. I'm not sure how useful a mock API would be to an orchestration tool like Flintrock.

You can read about how Flintrock's tests are setup here. Some tests run locally and some, if explicitly enabled, will run on your AWS account.

nchammas commented 6 years ago

Closing this as a duplicate of #82, per my earlier comment.