SBT doesn't recognize dbcCreateCluster as a valid key

[the-jackalope]

I have the sbt-databricks plugin installed in my project and am able to use most of the functionality in it such as dbcDeploy, dbcListClusters, etc. However sbt does not recognize dbcCreateCluster and gives the generic error:

[error] Not a valid command: dbcCreateCluster
[error] Not a valid key: dbcCreateCluster (similar: dbcRestartClusters, dbcFetchClusters, dbcClusters)
[error] dbcCreateCluster

dbcCreateCluster is listed as a valid command in the MD file so I was wondering if this is still the case? Any help would be appreciated.

Thanks!

[brkyvz]

Hi @the-jackalope , would you mind sharing the version of sbt-databricks that you are using?

[the-jackalope]

@brkyvz we're using 0.1.5

[brkyvz]

@the-jackalope Are you getting this error when using the sbt console? Could you also share your build file please?

[the-jackalope]

@brkyvz it happens both when I call from the console and when I'm developing in IntelliJ. In IntelliJ it doesn't recognize the command and says it's an unrecognized symbol. Here's my build.sbt:

// The following variables affect how releases go out with `sbt dbcDeploy`
val IS_RELEASE              = false
val PROJECT_VERSION         = "0.5.0" + (if(IS_RELEASE) "" else "-SNAPSHOT")
val BETA_CLUSTER_NAME       = "my-cluster"
val TARGET_CLUSTER: String  = if (!IS_RELEASE) BETA_CLUSTER_NAME else "ALL_CLUSTERS"
val TARGET_LIBRARY_PATH     = "/Shared/Libraries"

val PROJECT_NAME            = ""
val PROJECT_ORGANIZATION    = ""
val SCALA_VERSION           = "2.11.8"
val MAIN_CLASS              = ""

///////////////////////////////////////////////////////////////////////////////

name         := PROJECT_NAME
organization := PROJECT_ORGANIZATION
version      := PROJECT_VERSION
scalaVersion := SCALA_VERSION
exportJars   := true

mainClass in (Compile, run) := Some(MAIN_CLASS) // run with `sbt run`

def settingFromENV(settingName: String): String = sys.env.getOrElse(settingName, "")

dbcUsername         := settingFromENV("DATABRICKS_USERNAME")
dbcPassword         := settingFromENV("DATABRICKS_PASSWORD")
dbcApiUrl           := settingFromENV("DATABRICKS_URL")
dbcLibraryPath      := TARGET_LIBRARY_PATH
dbcClusters         := Seq(TARGET_CLUSTER)
dbcRestartOnAttach  := !IS_RELEASE

// Commented out til we find out what's up with dbcCreateCluster

//dbcNumWorkerContainers := num  // Integer: The desired size of the cluster (in worker containers).
//dbcSpotInstance := // Boolean for choosing whether to use Spot or On-Demand instances
//dbcSparkVersion := // String: The Spark version to be used e.g. "1.6.x"
//dbcZoneId := "aws-zone-here"// String: AWS zone e.g. ap-southeast-2

///////////////////////////////////////////////////////////////////////////////

resolvers           ++= Resolvers.resolverList

libraryDependencies ++= Dependencies.dependencyList

///////////////////////////////////////////////////////////////////////////////
///////////////////////////////////////////////////////////////////////////////

lazy val preDeploy      = taskKey[Unit]("Prepares for a deploy to Databricks")
lazy val hasCredentials = taskKey[Unit]("Checks for the user's credentials")
lazy val clusterExists  = taskKey[Unit]("Checks if the cluster exists")

// Bug in the sbt-databricks plugin - this doesn't currently work...
//lazy val createCluster  = taskKey[Unit]("Creates a cluster if none exist")

hasCredentials := {
  println("Checking environment variables...")
  val envKeys = Seq("DATABRICKS_USERNAME", "DATABRICKS_PASSWORD", "DATABRICKS_URL")
  val missingKeys = (envKeys map settingFromENV) map (_ == "")
  if (missingKeys reduce (_ || _)) {
    println("Error - missing Databricks secrets in environment. Export these before deploying")
    println("Exiting...")
    sys.exit(1)
  } else {
    println("Environment variables OK.")
  }
}

clusterExists := {
  println("Searching for cluster <dev-cluster-name>.")
  val stream: Stream[String] = ("sbt dbcListClusters | grep 14" lines_!)
  val exists = stream exists (cluster => cluster.contains("Running") && cluster.contains("<default-cluster-name>"))
  if(!exists) {
    println("Cluster does not exist. Please create the cluster before deploying.")
    println("Exiting...")
    sys.exit(1)
  } else println("Cluster found. Starting deploy process...")
}

val dynamic = Def.sequential(
  hasCredentials,
  clusterExists
)

preDeploy := dynamic.value

dbcDeploy := {
  preDeploy.value
  dbcDeploy.value
}

///////////////////////////////////////////////////////////////////////////////
///////////////////////////////////////////////////////////////////////////////

//// Strips fixture data from test artifact:
//
def fixtureFilter(path: String): Boolean =  path.contains("fixtures")

mappings in (Test, packageBin) ~= {
  _.filter { case (file, _) => ! fixtureFilter(file.getCanonicalPath) }
}

///////////////////////////////////////////////////////////////////////////////

and my plugins.sbt:

logLevel := sbt.Level.Warn

addSbtPlugin("com.twitter" % "scrooge-sbt-plugin" % "4.10.0")

// First-party SBT plugin to deploy and interact with the Databricks REST API
addSbtPlugin("com.databricks" %% "sbt-databricks" % "0.1.5")

// For better Jetbrains IntelliJ integration
addSbtPlugin("com.github.mpeltonen" % "sbt-idea" % "1.6.0")

addSbtPlugin("net.virtual-void" % "sbt-dependency-graph" % "0.8.2")

// For checking newer versions of dependencies : https://github.com/rtimush/sbt-updates
addSbtPlugin("com.timushev.sbt" % "sbt-updates" % "0.2.0")

Thanks! Really appreciate the help.

[the-jackalope]

Hi @brkyvz - I know you're a busy guy but just wondering if you'd learned anything more about this?

Thanks!