I've compiled and configured SAC and when executing a simple scala app I'm getting the error below.
I'm using Spark 2.4.7 with Scala 2.11.12
The scala app starts ok, but the error is shown after this first SQL command:
"spark.sql("CREATE DATABASE IF NOT EXISTS Demo")"
If that line is commented then the executions ends ok.
21/02/20 02:09:40 INFO StateStoreCoordinatorRef: Registered StateStoreCoordinator endpoint
Exception in thread "main" java.lang.IllegalArgumentException: Error while instantiating 'org.apache.spark.sql.internal.SessionStateBuilder':
at org.apache.spark.sql.SparkSession$.org$apache$spark$sql$SparkSession$$instantiateSessionState(SparkSession.scala:1118)
at org.apache.spark.sql.SparkSession$$anonfun$sessionState$2.apply(SparkSession.scala:145)
at org.apache.spark.sql.SparkSession$$anonfun$sessionState$2.apply(SparkSession.scala:144)
at scala.Option.getOrElse(Option.scala:121)
at org.apache.spark.sql.SparkSession.sessionState$lzycompute(SparkSession.scala:144)
at org.apache.spark.sql.SparkSession.sessionState(SparkSession.scala:141)
at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:643)
at org.apache.spark.sql.SQLContext.sql(SQLContext.scala:694)
at sparksqldemo$.main(sparksqldemo.scala:15)
at sparksqldemo.main(sparksqldemo.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:845)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:161)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:184)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:920)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:929)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: org.apache.spark.SparkException: Exception when registering StreamingQueryListener
at org.apache.spark.sql.streaming.StreamingQueryManager.(StreamingQueryManager.scala:71)
at org.apache.spark.sql.internal.BaseSessionStateBuilder.streamingQueryManager(BaseSessionStateBuilder.scala:260)
at org.apache.spark.sql.internal.BaseSessionStateBuilder.build(BaseSessionStateBuilder.scala:296)
at org.apache.spark.sql.SparkSession$.org$apache$spark$sql$SparkSession$$instantiateSessionState(SparkSession.scala:1115)
... 21 more
Caused by: java.lang.IllegalArgumentException: requirement failed: com.hortonworks.spark.atlas.SparkAtlasEventTracker is not a subclass of org.apache.spark.sql.streaming.StreamingQueryListener.
at scala.Predef$.require(Predef.scala:224)
at org.apache.spark.util.Utils$$anonfun$loadExtensions$1.apply(Utils.scala:2683)
at org.apache.spark.util.Utils$$anonfun$loadExtensions$1.apply(Utils.scala:2680)
at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
at scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:241)
at scala.collection.AbstractTraversable.flatMap(Traversable.scala:104)
at org.apache.spark.util.Utils$.loadExtensions(Utils.scala:2680)
at org.apache.spark.sql.streaming.StreamingQueryManager$$anonfun$1.apply(StreamingQueryManager.scala:63)
at org.apache.spark.sql.streaming.StreamingQueryManager$$anonfun$1.apply(StreamingQueryManager.scala:62)
at scala.Option.foreach(Option.scala:257)
at org.apache.spark.sql.streaming.StreamingQueryManager.(StreamingQueryManager.scala:62)
... 24 more
ramonbgc
commented
3 years ago
Hi!
I've compiled and configured SAC and when executing a simple scala app I'm getting the error below. I'm using Spark 2.4.7 with Scala 2.11.12 The scala app starts ok, but the error is shown after this first SQL command: "spark.sql("CREATE DATABASE IF NOT EXISTS Demo")"
If that line is commented then the executions ends ok.
21/02/20 02:09:40 INFO StateStoreCoordinatorRef: Registered StateStoreCoordinator endpoint Exception in thread "main" java.lang.IllegalArgumentException: Error while instantiating 'org.apache.spark.sql.internal.SessionStateBuilder': at org.apache.spark.sql.SparkSession$.org$apache$spark$sql$SparkSession$$instantiateSessionState(SparkSession.scala:1118) at org.apache.spark.sql.SparkSession$$anonfun$sessionState$2.apply(SparkSession.scala:145) at org.apache.spark.sql.SparkSession$$anonfun$sessionState$2.apply(SparkSession.scala:144) at scala.Option.getOrElse(Option.scala:121) at org.apache.spark.sql.SparkSession.sessionState$lzycompute(SparkSession.scala:144) at org.apache.spark.sql.SparkSession.sessionState(SparkSession.scala:141) at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:643) at org.apache.spark.sql.SQLContext.sql(SQLContext.scala:694) at sparksqldemo$.main(sparksqldemo.scala:15) at sparksqldemo.main(sparksqldemo.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52) at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:845) at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:161) at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:184) at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86) at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:920) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:929) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) Caused by: org.apache.spark.SparkException: Exception when registering StreamingQueryListener at org.apache.spark.sql.streaming.StreamingQueryManager.(StreamingQueryManager.scala:71)
at org.apache.spark.sql.internal.BaseSessionStateBuilder.streamingQueryManager(BaseSessionStateBuilder.scala:260)
at org.apache.spark.sql.internal.BaseSessionStateBuilder.build(BaseSessionStateBuilder.scala:296)
at org.apache.spark.sql.SparkSession$.org$apache$spark$sql$SparkSession$$instantiateSessionState(SparkSession.scala:1115)
... 21 more
Caused by: java.lang.IllegalArgumentException: requirement failed: com.hortonworks.spark.atlas.SparkAtlasEventTracker is not a subclass of org.apache.spark.sql.streaming.StreamingQueryListener.
at scala.Predef$.require(Predef.scala:224)
at org.apache.spark.util.Utils$$anonfun$loadExtensions$1.apply(Utils.scala:2683)
at org.apache.spark.util.Utils$$anonfun$loadExtensions$1.apply(Utils.scala:2680)
at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
at scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:241)
at scala.collection.AbstractTraversable.flatMap(Traversable.scala:104)
at org.apache.spark.util.Utils$.loadExtensions(Utils.scala:2680)
at org.apache.spark.sql.streaming.StreamingQueryManager$$anonfun$1.apply(StreamingQueryManager.scala:63)
at org.apache.spark.sql.streaming.StreamingQueryManager$$anonfun$1.apply(StreamingQueryManager.scala:62)
at scala.Option.foreach(Option.scala:257)
at org.apache.spark.sql.streaming.StreamingQueryManager.(StreamingQueryManager.scala:62)
... 24 more