search

hortonworks-spark / spark-atlas-connector

A Spark Atlas connector to track data lineage in Apache Atlas
Apache License 2.0
264 stars 149 forks source link

Compile is failed with Spark 3.1.1 and Scala 2.12.10 #317

Open picomy opened 3 years ago

picomy commented 3 years ago

Dear team, Today, I compile the package against Spark 3.1.1 and Scala 2.12.10, the result is failed. The error messages are the following

[ERROR] /Users/picomy/Downloads/spark-atlas-connector/spark-atlas-connector/src/main/scala/com/hortonworks/spark/atlas/sql/CommandsHarvester.scala:34: object DataSourceV2ScanExec is not a member of package org.apache.spark.sql.execution.datasources.v2
[ERROR] /Users/picomy/Downloads/spark-atlas-connector/spark-atlas-connector/src/main/scala/com/hortonworks/spark/atlas/sql/CommandsHarvester.scala:35: object MicroBatchWriter is not a member of package org.apache.spark.sql.execution.streaming.sources
[ERROR] /Users/picomy/Downloads/spark-atlas-connector/spark-atlas-connector/src/main/scala/com/hortonworks/spark/atlas/sql/CommandsHarvester.scala:36: object v2 is not a member of package org.apache.spark.sql.sources
[ERROR] /Users/picomy/Downloads/spark-atlas-connector/spark-atlas-connector/src/main/scala/com/hortonworks/spark/atlas/sql/CommandsHarvester.scala:144: value tableIdentifier is not a member of org.apache.spark.sql.catalyst.analysis.UnresolvedRelation
[ERROR] /Users/picomy/Downloads/spark-atlas-connector/spark-atlas-connector/src/main/scala/com/hortonworks/spark/atlas/sql/CommandsHarvester.scala:567: not found: type DataSourceWriter
[ERROR] /Users/picomy/Downloads/spark-atlas-connector/spark-atlas-connector/src/main/scala/com/hortonworks/spark/atlas/sql/CommandsHarvester.scala:193: value writer is not a member of org.apache.spark.sql.execution.datasources.v2.WriteToDataSourceV2Exec
[ERROR] /Users/picomy/Downloads/spark-atlas-connector/spark-atlas-connector/src/main/scala/com/hortonworks/spark/atlas/sql/CommandsHarvester.scala:194: not found: type MicroBatchWriter
[ERROR] /Users/picomy/Downloads/spark-atlas-connector/spark-atlas-connector/src/main/scala/com/hortonworks/spark/atlas/sql/CommandsHarvester.scala:295: not found: type DataSourceWriter
[ERROR] /Users/picomy/Downloads/spark-atlas-connector/spark-atlas-connector/src/main/scala/com/hortonworks/spark/atlas/sql/CommandsHarvester.scala:400: not found: type DataSourceWriter
[ERROR] /Users/picomy/Downloads/spark-atlas-connector/spark-atlas-connector/src/main/scala/com/hortonworks/spark/atlas/sql/CommandsHarvester.scala:430: not found: type DataSourceWriter
[ERROR] /Users/picomy/Downloads/spark-atlas-connector/spark-atlas-connector/src/main/scala/com/hortonworks/spark/atlas/sql/CommandsHarvester.scala:313: value writer is not a member of org.apache.spark.sql.execution.datasources.v2.WriteToDataSourceV2Exec
[ERROR] /Users/picomy/Downloads/spark-atlas-connector/spark-atlas-connector/src/main/scala/com/hortonworks/spark/atlas/sql/CommandsHarvester.scala:323: type mismatch;
 found   : Seq[Any]
 required: Seq[com.hortonworks.spark.atlas.SACAtlasReferenceable]
[ERROR] /Users/picomy/Downloads/spark-atlas-connector/spark-atlas-connector/src/main/scala/com/hortonworks/spark/atlas/sql/CommandsHarvester.scala:326: type mismatch;
 found   : Seq[Any]
 required: Seq[com.hortonworks.spark.atlas.SACAtlasReferenceable]
[ERROR] /Users/picomy/Downloads/spark-atlas-connector/spark-atlas-connector/src/main/scala/com/hortonworks/spark/atlas/sql/CommandsHarvester.scala:366: not found: type DataSourceWriter
[ERROR] /Users/picomy/Downloads/spark-atlas-connector/spark-atlas-connector/src/main/scala/com/hortonworks/spark/atlas/sql/CommandsHarvester.scala:367: not found: type DataSourceWriter
[ERROR] /Users/picomy/Downloads/spark-atlas-connector/spark-atlas-connector/src/main/scala/com/hortonworks/spark/atlas/sql/CommandsHarvester.scala:370: not found: type DataSourceWriter
[ERROR] /Users/picomy/Downloads/spark-atlas-connector/spark-atlas-connector/src/main/scala/com/hortonworks/spark/atlas/sql/CommandsHarvester.scala:373: not found: type DataSourceWriter
[ERROR] /Users/picomy/Downloads/spark-atlas-connector/spark-atlas-connector/src/main/scala/com/hortonworks/spark/atlas/sql/CommandsHarvester.scala:376: not found: type MicroBatchWriter
[ERROR] /Users/picomy/Downloads/spark-atlas-connector/spark-atlas-connector/src/main/scala/com/hortonworks/spark/atlas/sql/CommandsHarvester.scala:380: not found: type DataSourceWriter
[ERROR] /Users/picomy/Downloads/spark-atlas-connector/spark-atlas-connector/src/main/scala/com/hortonworks/spark/atlas/sql/CommandsHarvester.scala:388: value source is not a member of org.apache.spark.sql.execution.datasources.v2.DataSourceV2Relation
[ERROR] /Users/picomy/Downloads/spark-atlas-connector/spark-atlas-connector/src/main/scala/com/hortonworks/spark/atlas/sql/CommandsHarvester.scala:394: not found: type DataSourceV2ScanExec
[ERROR] /Users/picomy/Downloads/spark-atlas-connector/spark-atlas-connector/src/main/scala/com/hortonworks/spark/atlas/sql/CommandsHarvester.scala:395: value source is not a member of org.apache.spark.sql.execution.SparkPlan
[ERROR] /Users/picomy/Downloads/spark-atlas-connector/spark-atlas-connector/src/main/scala/com/hortonworks/spark/atlas/sql/CommandsHarvester.scala:396: value options is not a member of org.apache.spark.sql.execution.SparkPlan
[ERROR] /Users/picomy/Downloads/spark-atlas-connector/spark-atlas-connector/src/main/scala/com/hortonworks/spark/atlas/sql/CommandsHarvester.scala:414: value tableIdentifier is not a member of org.apache.spark.sql.catalyst.analysis.UnresolvedRelation
[ERROR] /Users/picomy/Downloads/spark-atlas-connector/spark-atlas-connector/src/main/scala/com/hortonworks/spark/atlas/sql/CommandsHarvester.scala:416: value tableIdentifier is not a member of org.apache.spark.sql.catalyst.analysis.UnresolvedRelation
[ERROR] /Users/picomy/Downloads/spark-atlas-connector/spark-atlas-connector/src/main/scala/com/hortonworks/spark/atlas/sql/CommandsHarvester.scala:461: not found: type MicroBatchWriter
[ERROR] /Users/picomy/Downloads/spark-atlas-connector/spark-atlas-connector/src/main/scala/com/hortonworks/spark/atlas/sql/CommandsHarvester.scala:464: not found: type DataSourceWriter
[ERROR] /Users/picomy/Downloads/spark-atlas-connector/spark-atlas-connector/src/main/scala/org/apache/spark/sql/kafka010/atlas/ExtractFromDataSource.scala:39: object v2 is not a member of package org.apache.spark.sql.sources
[ERROR] /Users/picomy/Downloads/spark-atlas-connector/spark-atlas-connector/src/main/scala/org/apache/spark/sql/kafka010/atlas/ExtractFromDataSource.scala:182: not found: type DataSourceV2
[ERROR] /Users/picomy/Downloads/spark-atlas-connector/spark-atlas-connector/src/main/scala/com/hortonworks/spark/atlas/sql/CommandsHarvester.scala:559: not found: type DataSourceV2ScanExec
[ERROR] /Users/picomy/Downloads/spark-atlas-connector/spark-atlas-connector/src/main/scala/org/apache/spark/sql/kafka010/atlas/ExtractFromDataSource.scala:30: object DataSourceV2ScanExec is not a member of package org.apache.spark.sql.execution.datasources.v2
[ERROR] /Users/picomy/Downloads/spark-atlas-connector/spark-atlas-connector/src/main/scala/org/apache/spark/sql/kafka010/atlas/ExtractFromDataSource.scala:89: not found: type DataSourceV2ScanExec
[ERROR] /Users/picomy/Downloads/spark-atlas-connector/spark-atlas-connector/src/main/scala/com/hortonworks/spark/atlas/sql/CommandsHarvester.scala:568: not found: type MicroBatchWriter
[ERROR] /Users/picomy/Downloads/spark-atlas-connector/spark-atlas-connector/src/main/scala/org/apache/spark/sql/kafka010/atlas/ExtractFromDataSource.scala:31: object MicroBatchWriter is not a member of package org.apache.spark.sql.execution.streaming.sources
[ERROR] /Users/picomy/Downloads/spark-atlas-connector/spark-atlas-connector/src/main/scala/org/apache/spark/sql/kafka010/atlas/ExtractFromDataSource.scala:66: not found: type MicroBatchWriter
[ERROR] /Users/picomy/Downloads/spark-atlas-connector/spark-atlas-connector/src/main/scala/com/hortonworks/spark/atlas/sql/SparkExecutionPlanProcessor.scala:29: object MicroBatchWriter is not a member of package org.apache.spark.sql.execution.streaming.sources
[ERROR] /Users/picomy/Downloads/spark-atlas-connector/spark-atlas-connector/src/main/scala/com/hortonworks/spark/atlas/sql/SparkExecutionPlanProcessor.scala:31: object v2 is not a member of package org.apache.spark.sql.sources
[ERROR] /Users/picomy/Downloads/spark-atlas-connector/spark-atlas-connector/src/main/scala/com/hortonworks/spark/atlas/sql/SparkExecutionPlanProcessor.scala:38: object v2 is not a member of package org.apache.spark.sql.sources
[ERROR] /Users/picomy/Downloads/spark-atlas-connector/spark-atlas-connector/src/main/scala/com/hortonworks/spark/atlas/sql/SparkExecutionPlanProcessor.scala:173: not found: type StreamWriter
[ERROR] /Users/picomy/Downloads/spark-atlas-connector/spark-atlas-connector/src/main/scala/com/hortonworks/spark/atlas/sql/SparkExecutionPlanProcessor.scala:83: value ++= is not a member of Seq[org.apache.spark.sql.execution.SparkPlan]
  Expression does not convert to assignment because:
    not found: type MicroBatchWriter
    expansion: outNodes = outNodes.++(Seq(WriteToDataSourceV2Exec(new <MicroBatchWriter: error>(0, new SinkDataSourceWriter(sink)), qd.qe.sparkPlan)))
[ERROR] /Users/picomy/Downloads/spark-atlas-connector/spark-atlas-connector/src/main/scala/com/hortonworks/spark/atlas/sql/SparkExecutionPlanProcessor.scala:100: not found: value PersistedView
[ERROR] /Users/picomy/Downloads/spark-atlas-connector/spark-atlas-connector/src/main/scala/com/hortonworks/spark/atlas/sql/SparkExecutionPlanProcessor.scala:174: not found: type DataWriterFactory
[ERROR] /Users/picomy/Downloads/spark-atlas-connector/spark-atlas-connector/src/main/scala/com/hortonworks/spark/atlas/sql/SparkExecutionPlanProcessor.scala:177: not found: type WriterCommitMessage
[ERROR] /Users/picomy/Downloads/spark-atlas-connector/spark-atlas-connector/src/main/scala/com/hortonworks/spark/atlas/sql/SparkExecutionPlanProcessor.scala:180: not found: type WriterCommitMessage
[ERROR] /Users/picomy/Downloads/spark-atlas-connector/spark-atlas-connector/src/main/scala/com/hortonworks/spark/atlas/sql/SparkExecutionPlanProcessor.scala:183: not found: type WriterCommitMessage
[ERROR] /Users/picomy/Downloads/spark-atlas-connector/spark-atlas-connector/src/main/scala/com/hortonworks/spark/atlas/sql/SparkExecutionPlanProcessor.scala:186: not found: type WriterCommitMessage
[ERROR] /Users/picomy/Downloads/spark-atlas-connector/spark-atlas-connector/src/main/scala/com/hortonworks/spark/atlas/utils/SparkUtils.scala:212: value getExecutionList is not a member of org.apache.spark.sql.hive.thriftserver.ui.HiveThriftServer2Listener
[ERROR] /Users/picomy/Downloads/spark-atlas-connector/spark-atlas-connector/src/main/scala/org/apache/spark/sql/kafka010/atlas/ExtractFromDataSource.scala:32: object v2 is not a member of package org.apache.spark.sql.sources
[ERROR] /Users/picomy/Downloads/spark-atlas-connector/spark-atlas-connector/src/main/scala/org/apache/spark/sql/kafka010/atlas/ExtractFromDataSource.scala:33: object v2 is not a member of package org.apache.spark.sql.sources
[ERROR] /Users/picomy/Downloads/spark-atlas-connector/spark-atlas-connector/src/main/scala/org/apache/spark/sql/kafka010/atlas/ExtractFromDataSource.scala:233: not found: type DataWriterFactory
[ERROR] /Users/picomy/Downloads/spark-atlas-connector/spark-atlas-connector/src/main/scala/org/apache/spark/sql/kafka010/atlas/ExtractFromDataSource.scala:197: not found: type DataWriterFactory
[ERROR] /Users/picomy/Downloads/spark-atlas-connector/spark-atlas-connector/src/main/scala/org/apache/spark/sql/kafka010/atlas/ExtractFromDataSource.scala:213: not found: type InputPartition
[ERROR] /Users/picomy/Downloads/spark-atlas-connector/spark-atlas-connector/src/main/scala/org/apache/spark/sql/kafka010/atlas/ExtractFromDataSource.scala:217: not found: type InputPartition
[WARNING] /Users/picomy/Downloads/spark-atlas-connector/spark-atlas-connector/src/main/scala/org/apache/spark/sql/kafka010/atlas/ExtractFromDataSource.scala:244: non-variable type argument String in type pattern java.util.Map[String,Object] is unchecked since it is eliminated by erasure
[WARNING] /Users/picomy/Downloads/spark-atlas-connector/spark-atlas-connector/src/main/scala/org/apache/spark/sql/kafka010/atlas/ExtractFromDataSource.scala:247: non-variable type argument String in type pattern scala.collection.immutable.Map[String,String] (the underlying of Map[String,String]) is unchecked since it is eliminated by erasure
[ERROR] /Users/picomy/Downloads/spark-atlas-connector/spark-atlas-connector/src/main/scala/org/apache/spark/sql/kafka010/atlas/ExtractFromDataSource.scala:309: not found: type InputPartition
[ERROR] /Users/picomy/Downloads/spark-atlas-connector/spark-atlas-connector/src/main/scala/org/apache/spark/sql/kafka010/atlas/ExtractFromDataSource.scala:335: not found: type InputPartition
[WARNING] two warnings found
[ERROR] 55 errors found
ailsexixi commented 2 years ago

I have encountered the same problem. Is there any solution

SmaatoDataTeam commented 1 year ago

same to me