Master is put on a huge instance when you need a huge slave

nchammas / flintrock

A command-line tool for launching Apache Spark clusters.

Apache License 2.0

636 stars 116 forks source link

Master is put on a huge instance when you need a huge slave #284

Closed pferrel closed 5 years ago

pferrel commented 5 years ago

Flintrock version: 0.11.0
Python version: 3.7
OS: macOS Mojave

It seem like there is only one instance size so the master is started on what can often be a huge instance. If you need a huge slave, the master gets the same instance even though the master does very little and needs only a tiny instance.

Is there any way around this?

nchammas commented 5 years ago

Duplicate of #199.

Not sure if there is a workaround available. You can try launching a cluster with 1 instance type and then adding new slaves with a different instance type, but I don't know if Flintrock currently allows that.