search

databricks / Spark-The-Definitive-Guide

Spark: The Definitive Guide's Code Repository
http://shop.oreilly.com/product/0636920034957.do
Other
2.85k stars 2.76k forks source link

Chapter 3: java.lang.IllegalStateException: Cannot call methods on a stopped SparkContext. #50

Open kumgaurav opened 4 years ago

kumgaurav commented 4 years ago

Created a scala program and trying to run the streaming code. it is filing with below error

19/11/09 07:25:24 ERROR StreamExecution: Query customer_purchases_2 [id = 6863c8d1-fd1c-49eb-a454-92825f0d2782, runId = 8adbd1e6-04e1-4b79-9fad-05950f359e77] terminated with error java.lang.IllegalStateException: Cannot call methods on a stopped SparkContext. This stopped SparkContext was created at:

org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:901) spark_guide.chapter3.StructuredStreaming$.delayedEndpoint$spark_guide$chapter3$StructuredStreaming$1(StructuredStreaming.scala:12) spark_guide.chapter3.StructuredStreaming$delayedInit$body.apply(StructuredStreaming.scala:9) scala.Function0$class.apply$mcV$sp(Function0.scala:34) scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12) scala.App$$anonfun$main$1.apply(App.scala:76) scala.App$$anonfun$main$1.apply(App.scala:76) scala.collection.immutable.List.foreach(List.scala:392) scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:35) scala.App$class.main(App.scala:76) spark_guide.chapter3.StructuredStreaming$.main(StructuredStreaming.scala:9) spark_guide.chapter3.StructuredStreaming.main(StructuredStreaming.scala)

WilliamCVan commented 3 years ago

To fix that bug add this line after the writeStream.start() command, it will keep the sparkContext open for you to see the results. I was writing to console, was not using in memory table fyi.

spark.streams.awaitAnyTermination()