search

hortonworks-spark / spark-atlas-connector

A Spark Atlas connector to track data lineage in Apache Atlas
Apache License 2.0
263 stars 149 forks source link

SAC does not work #316

Closed ighack closed 2 years ago

ighack commented 2 years ago

atlas 2.1.0 CDH 6.3.2 spark doesn't have thriftserver so, I modify pom.xml 

<dependency>
      <groupId>org.apache.spark</groupId>
      <artifactId>spark-hive-thriftserver_${scala.binary.version}</artifactId>
      <version>2.4.0</version>
      <!--<scope>provided</scope>-->
    </dependency>

spark-shell --jars hdfs://nameservice1/sparklib/* --files /hadoop/apache-atlas-2.1.0/conf/atlas-application.properties --conf spark.extraListeners=com.hortonworks.spark.atlas.SparkAtlasEventTracker --conf spark.sql.queryExecutionListeners=com.hortonworks.spark.atlas.SparkAtlasEventTracker --conf spark.sql.streaming.streamingQueryListeners=com.hortonworks.spark.atlas.SparkAtlasStreamingQueryEventTracker

I get

WARN sql.CommandsHarvester$: Missing unknown leaf node: LocalRelation , [id#0, name#1, year#2, city#3]

ighack commented 2 years ago

SAC doesn't hook create