search

tupol / spark-utils

Basic framework utilities to quickly start writing production ready Apache Spark applications
MIT License
36 stars 6 forks source link

SparkApp fails if no configuration is expected #9

Closed tupol closed 5 years ago

tupol commented 5 years ago

When implementing a SparkApp that has no configuration expectations, the initialization of the app fails with the following exception:

An unexpected com.typesafe.config.ConfigException$Missing was thrown.
ScalaTestFailureLocation: org.tupol.spark.SparkAppSpec$$anonfun$5 at (SparkAppSpec.scala:64)
org.scalatest.exceptions.TestFailedException: An unexpected com.typesafe.config.ConfigException$Missing was thrown.
. . .

Sample test for SparkApp:

  object MockAppNoConfig extends SparkApp[String, Unit] {
    def createContext(config: Config): String = "Hello"
    override def run(implicit spark: SparkSession, config: String): Unit = Unit
  }

  test("SparkApp.main successfully completes") {
    noException shouldBe thrownBy(MockAppNoConfig.main(Array()))
  }