search

pishen / sbt-lighter

SBT plugin for Apache Spark on AWS EMR
Apache License 2.0
57 stars 15 forks source link

adding support for ec2KeyName #26

Closed alvinhenrick closed 7 years ago

alvinhenrick commented 7 years ago

@pishen Adding support for optional attribute ec2KeyName

alvinhenrick commented 7 years ago

@pishen Could you please have a look and see if it's possible to merge ?

pishen commented 7 years ago

Could this be done by

sparkJobFlowInstancesConfig := sparkJobFlowInstancesConfig.value.withEc2KeyName("your-keypair")

as in README?

pishen commented 7 years ago

@alvinhenrick I think we can provide this feature. Before I merge, could you change the setting key from ec2KeyName to sparkEc2KeyName?

alvinhenrick commented 7 years ago

@pishen Done renamed the variable to sparkEc2KeyName I think the way you have implemented is awesome flexibility to allow user to override the settings but I thought it would be good for plugin to support most of the things out of the box so user have to make minimum changes to confgure and it's a build tool plugin.

Thanks! Warm Regards, Alvin.

pishen commented 7 years ago

@alvinhenrick Actually, I am not sure what is the definition of "most of the things" here. Everyone has his/her own requirement when running EMR. And it may not be suitable for us to wrap all the functions from Java API, sometimes just to make the line a little bit shorter?

It's still welcome. But I think there're something with higher priority. For example, we may figure out how to write some tests for this plugin when it's growing, or how to handle the cluster name collision problem.