search

hortonworks-spark / spark-atlas-connector

A Spark Atlas connector to track data lineage in Apache Atlas
Apache License 2.0
263 stars 149 forks source link

java.lang.NoSuchMethodError: org.apache.spark.sql.execution.QueryExecution.id()J #311

Closed GHOSTHSP closed 3 years ago

GHOSTHSP commented 3 years ago

Exception in thread "main" java.lang.NoSuchMethodError: org.apache.spark.sql.execution.QueryExecution.id()J at com.hortonworks.spark.atlas.SparkAtlasEventTracker.onSuccess(SparkAtlasEventTracker.scala:83) at org.apache.spark.sql.util.ExecutionListenerManager$$anonfun$onSuccess$1$$anonfun$apply$mcV$sp$1.apply(QueryExecutionListener.scala:127) at org.apache.spark.sql.util.ExecutionListenerManager$$anonfun$onSuccess$1$$anonfun$apply$mcV$sp$1.apply(QueryExecutionListener.scala:126) at org.apache.spark.sql.util.ExecutionListenerManager$$anonfun$org$apache$spark$sql$util$ExecutionListenerManager$$withErrorHandling$1.apply(QueryExecutionListener.scala:148) at org.apache.spark.sql.util.ExecutionListenerManager$$anonfun$org$apache$spark$sql$util$ExecutionListenerManager$$withErrorHandling$1.apply(QueryExecutionListener.scala:146) at scala.collection.immutable.List.foreach(List.scala:392) at scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:35) at scala.collection.mutable.ListBuffer.foreach(ListBuffer.scala:45) at org.apache.spark.sql.util.ExecutionListenerManager.org$apache$spark$sql$util$ExecutionListenerManager$$withErrorHandling(QueryExecutionListener.scala:146) at org.apache.spark.sql.util.ExecutionListenerManager$$anonfun$onSuccess$1.apply$mcV$sp(QueryExecutionListener.scala:126) at org.apache.spark.sql.util.ExecutionListenerManager$$anonfun$onSuccess$1.apply(QueryExecutionListener.scala:126) at org.apache.spark.sql.util.ExecutionListenerManager$$anonfun$onSuccess$1.apply(QueryExecutionListener.scala:126) at org.apache.spark.sql.util.ExecutionListenerManager.readLock(QueryExecutionListener.scala:159) at org.apache.spark.sql.util.ExecutionListenerManager.onSuccess(QueryExecutionListener.scala:125) at org.apache.spark.sql.DataFrameWriter.runCommand(DataFrameWriter.scala:678) at org.apache.spark.sql.DataFrameWriter.saveToV1Source(DataFrameWriter.scala:285) at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:271) at org.apache.spark.sql.DataFrameWriter.jdbc(DataFrameWriter.scala:515) at com.kubedata.insertdemo.HiveToMysql$.main(HiveToMysql.scala:26) at com.kubedata.insertdemo.HiveToMysql.main(HiveToMysql.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52) at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:845) at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:161) at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:184) at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86) at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:920) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:929) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

What is the reason ? How to solve the problem ? Thanks

pj001 commented 3 years ago

How is it solved? I have a similar problem

GHOSTHSP commented 3 years ago

Due to the local environment Spark and Spark-Atlas-Connector compile Spark version inconformity, You may have tried to change the Spark-Atlas-Connector pom.xml spark version to match your local environment, And recompile

                                                                                                                                                                                                                                                       Best Wishes

At 2021-03-15 14:19:54, "pj001" @.***> wrote:

How is it solved? I have a similar problem

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub, or unsubscribe.