search

pishen / sbt-lighter

SBT plugin for Apache Spark on AWS EMR
Apache License 2.0
57 stars 15 forks source link

Skipping sbt in sbt-emr-spark? #15

Open deeplook opened 7 years ago

deeplook commented 7 years ago

This is an interesting plugin, but I wonder if the functionality it provides would not be even more interesting outside of sbt? Then it would be very likely much easier to reuse, e.g. in a CI/CD context.

pishen commented 7 years ago

@deeplook Do you mean making the part that calls EMR's API into a Scala library? Or what usage scenario do you expect?

I use sbt-assembly to generate the jar file, and integrate the plugin's configurations into build.sbt instead of handling another configuration file by myself. I think these are the main dependencies on sbt now. But any suggestions or ideas are welcomed.

deeplook commented 7 years ago

I've stumbled over this repo when looking for a tool to configure and launch EMR clusters. While the AWS CLI tools are useful, they don't really provide any features for easily handling a larger set of options. So people usually still write mostly messy, non-reusable bash scripts accessing tons of environment variables. While I would like to use something higher level, I think sbt does not make configuring anything easier. And yes, I still think a tool for configuring and launching EMR clusters would be a very useful thing to have, but I don't see the value of providing that as an sbt plugin if the most obvious (if not only) connection is that to a build process where a JAR file gets copied into some S3 bucket and/or directly into an EMR cluster. I would simply like such a tool to be useful without sbt, too. But then, maybe I'm too much of a believer in the "one tool for one purpose" philosophy... If so, feel free to ignore. ;-)

pishen commented 7 years ago

Hmmm, the original purpose of this project was to create a command in sbt which can handle all the steps from compiling the code to locating and submit the generated jar file to a new configured cluster for me. Since I have to run sbt compile as my first step anyway, I tried to let sbt do more jobs for me in order not to handle any other scripts or tools or config files beside sbt. I'm sorry that this didn't match your use case. Right now I haven't come up with any good idea of how to make the tool you described. Will leave a comment here if I came across something.