search

pishen / sbt-lighter

SBT plugin for Apache Spark on AWS EMR
Apache License 2.0
57 stars 15 forks source link

Add BootstrapActions setting #34

Closed echeipesh closed 6 years ago

echeipesh commented 6 years ago

This change exposes the EMR Bootstrap Actions configuration, allows to specify them explicitly. I tested this through publishLocal and it appears to work with or without the args specified.

While writing this I realized that this is already possible by hooking into sparkRunJobFlowRequest but thats not at all obvious unless you're pretty familiar with the AWS SDK and not pretty either.

If this looks good I would like to add BootstrapAction!.fromFile(file) method that would trigger an upload of bootstrap script during sparkSubmit and sparkSubmitMain tasks in subsequent PR.

pishen commented 6 years ago

LGTM! About the explanation in README, I think it's not obvious by adding one line in "Use SBT's config to provide multiple setting combinations" section. Could you move the example to "Other available settings" part instead?

pishen commented 6 years ago

Released in sbt-lighter 1.2.0, with default emr release updated to 5.17.0.