search

pishen / sbt-lighter

SBT plugin for Apache Spark on AWS EMR
Apache License 2.0
57 stars 15 forks source link

local and EMR #11

Closed lolaclinton closed 7 years ago

lolaclinton commented 7 years ago

Is it possible to have both a local spark version and an EMR version in the same SBT file?

pishen commented 7 years ago

@lolaclinton I'm not sure what you mean? Spark version of your program could be defined by

libraryDependencies ++= Seq(
  "org.apache.spark" %% "spark-core" % "2.1.1" % "provided"
)

But since it's provided, it will use the Spark provided by EMR when the job is submitted, the Spark provided by EMR is determined by EMR's release label, which can be changed by

sparkEmrRelease := "emr-5.5.0"

in build.sbt.