cerveada
commented
2 years ago Where are you running this from? Seems like the provided dependencies are missing on the classpath. For example, in Idea, there is an option to add provided dependencies.
Closed scaladevspark closed 2 years ago
cerveada
commented
2 years ago Where are you running this from? Seems like the provided dependencies are missing on the classpath. For example, in Idea, there is an option to add provided dependencies.
scaladevspark
commented
2 years ago please disregard, one of our devs tried to run it with spark 3.1.2. so of course it would not run. Runs fine under spark 3.2.0. Thanks.
Using Spark 3.2.0 with Spark Structured streaming and trying to write to kafka with confluent schema registry via "za.co.absa" % "abris_2.12" % "6.0.0"
However, getting the exception below about a missing method. What might be the reason? Library versions mismatch or some dependency missing? Thanks a lot.
java.lang.NoSuchMethodError: org.apache.spark.sql.internal.SQLConf$.AVRO_REBASE_MODE_IN_WRITE() Lorg/apache/spark/internal/config/ConfigEntry;\n\tat org.apache.spark.sql.avro.AvroSerializer.(AvroSerializer.scala:56)
\n\tat org.apache.spark.sql.avro.AbrisAvroSerializer.(AbrisAvroSerializer.scala:30)
\n\tat za.co.absa.abris.avro.sql.CatalystDataToAvro.serializer$lzycompute(CatalystDataToAvro.scala:41)
\n\tat za.co.absa.abris.avro.sql.CatalystDataToAvro.serializer(CatalystDataToAvro.scala:40)
\n\tat za.co.absa.abris.avro.sql.CatalystDataToAvro.nullSafeEval(CatalystDataToAvro.scala:58)
\n\tat org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage4.processNext(Unknown Source)
\n\tat org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
\n\tat org.apache.spark.sql.execution.WholeStageCodegenExec$$anon$1.hasNext(WholeStageCodegenExec.scala:755)
\n\tat org.apache.spark.sql.kafka010.KafkaWriteTask.execute(KafkaWriteTask.scala:51)
\n\tat org.apache.spark.sql.kafka010.KafkaWriter$.$anonfun$write$2(KafkaWriter.scala:72)
\n\tat scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
\n\tat org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1439)
\n\tat org.apache.spark.sql.kafka010.KafkaWriter$.$anonfun$write$1(KafkaWriter.scala:73)
\n\tat org.apache.spark.sql.kafka010.KafkaWriter$.$anonfun$write$1$adapted(KafkaWriter.scala:70)
\n\tat org.apache.spark.rdd.RDD.$anonfun$foreachPartition$2(RDD.scala:1020)
\n\tat org.apache.spark.rdd.RDD.$anonfun$foreachPartition$2$adapted(RDD.scala:1020)
\n\tat org.apache.spark.SparkContext.$anonfun$runJob$5(SparkContext.scala:2236)
\n\tat org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
\n\tat org.apache.spark.scheduler.Task.run(Task.scala:131)
\n\tat org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:497)
\n\tat org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1439)
\n\tat org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:500)
\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
\n\tat java.lang.Thread.run(Thread.java:748)
My current dependencies: scalaVersion := "2.12.3"
lazy val Versions = new { val spark = "3.2.0" val jackson = "2.9.9" val scalatest = "3.0.4" }
libraryDependencies ++= Seq( "org.apache.spark" %% "spark-core" % Versions.spark % Provided , "org.apache.spark" %% "spark-sql" % Versions.spark % Provided , "com.typesafe" % "config" % "1.3.1" , "io.confluent" % "kafka-avro-serializer" % "5.5.0" , "org.apache.kafka" %% "kafka" % "3.0.0" , "org.apache.spark" %% "spark-avro" % "3.2.0" , "com.github.scopt" %% "scopt" % "3.7.0" , "com.fasterxml.jackson.core" % "jackson-databind" % Versions.jackson , "com.fasterxml.jackson.core" % "jackson-core" % Versions.jackson , "com.holdenkarau" %% "spark-testing-base" % "3.2.0_1.1.1" % Test , "org.scalatest" %% "scalatest" % Versions.scalatest % Test , "za.co.absa" % "abris_2.12" % "6.0.0" , "io.prometheus" % "simpleclient" % "0.9.0" , "io.prometheus" % "simpleclient_pushgateway" % "0.9.0" )