search

hortonworks-spark / spark-atlas-connector

A Spark Atlas connector to track data lineage in Apache Atlas
Apache License 2.0
263 stars 149 forks source link

Unable to set configuration directory #303

Closed ajs6f closed 3 years ago

ajs6f commented 4 years ago

I'm having much trouble trying to get SAC set up with HDP 3.1.5. I'm running into the problem shown in a YARN log below, and clearly SAC isn't able to find atlas-application.properties. But I have carefully made sure that a good copy of that file is in the Spark conf directory for every node in my cluster and I've tried any number of places in additional config to insert atlas.conf as a sys prop or ATLAS_CONF as appropriate. I'm submitting jobs via Livy and the Spark Thrift server, so perhaps that has something to do with it, but it's not clear to me how.

Has anyone seen something like this? Is there a known way to get SAC running on HDP 3.1.5 that I'm missing? Thanks for any advice!

Full log at bottom, happy to provide anything else useful. Please notice the section:

20/07/06 17:05:48 INFO ApplicationProperties: Looking for atlas-application.properties in classpath
20/07/06 17:05:48 INFO ApplicationProperties: Looking for /atlas-application.properties in classpath
20/07/06 17:05:48 INFO ApplicationProperties: Loading atlas-application.properties from null

Full log: 

SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/hdp/3.1.4.0-315/spark-atlas-connector/spark-atlas-connector-assembly-0.1.0.3.1.4.0-315.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/hadoop/yarn/local/filecache/16/spark2-hdp-yarn-archive.tar.gz/slf4j-log4j12-1.7.16.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/hadoop/yarn/local/filecache/16/spark2-hdp-yarn-archive.tar.gz/spark-atlas-connector-assembly-0.1.0.3.1.4.0-315.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/hdp/3.1.4.0-315/hadoop/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
20/07/06 17:05:39 INFO SignalUtils: Registered signal handler for TERM
20/07/06 17:05:39 INFO SignalUtils: Registered signal handler for HUP
20/07/06 17:05:39 INFO SignalUtils: Registered signal handler for INT
20/07/06 17:05:39 INFO SecurityManager: Changing view acls to: yarn,sorokaa
20/07/06 17:05:39 INFO SecurityManager: Changing modify acls to: yarn,sorokaa
20/07/06 17:05:39 INFO SecurityManager: Changing view acls groups to: 
20/07/06 17:05:39 INFO SecurityManager: Changing modify acls groups to: 
20/07/06 17:05:39 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(yarn, sorokaa); groups with view permissions: Set(); users  with modify permissions: Set(yarn, sorokaa); groups with modify permissions: Set()
20/07/06 17:05:40 DEBUG SecurityManager: Created SSL options for fs: SSLOptions{enabled=false, port=None, keyStore=None, keyStorePassword=None, trustStore=None, trustStorePassword=None, protocol=None, enabledAlgorithms=Set()}
20/07/06 17:05:40 DEBUG SparkHadoopUtil: creating UGI for user: sorokaa
20/07/06 17:05:40 INFO ApplicationMaster: Preparing Local resources
20/07/06 17:05:41 INFO ApplicationMaster: ApplicationAttemptId: appattempt_1594045625213_0020_000001
20/07/06 17:05:41 DEBUG ShutdownHookManager: Adding shutdown hook
20/07/06 17:05:41 INFO ApplicationMaster: Starting the user application in a separate Thread
20/07/06 17:05:41 INFO ApplicationMaster: Waiting for spark context initialization...
20/07/06 17:05:41 INFO RSCDriver: Connecting to: test05.cm.cluster:10000
20/07/06 17:05:41 INFO RSCDriver: Starting RPC server...
20/07/06 17:05:41 INFO RpcServer: Connected to the port 10000
20/07/06 17:05:41 WARN RSCConf: Your hostname, test25.cm.cluster, resolves to a loopback address, but we couldn't find any external IP address!
20/07/06 17:05:41 WARN RSCConf: Set livy.rsc.rpc.server.address if you need to bind to another address.
20/07/06 17:05:42 INFO RSCDriver: Received job request 025e183c-b89e-46c8-85a2-6de7b2ec077f
20/07/06 17:05:42 INFO RSCDriver: SparkContext not yet up, queueing job request.
20/07/06 17:05:46 INFO SparkEntries: Starting Spark context...
20/07/06 17:05:46 INFO SparkContext: Running Spark version 2.3.2.3.1.4.0-315
20/07/06 17:05:46 INFO SparkContext: Submitted application: livy-session-74
20/07/06 17:05:47 INFO SecurityManager: Changing view acls to: yarn,sorokaa
20/07/06 17:05:47 INFO SecurityManager: Changing modify acls to: yarn,sorokaa
20/07/06 17:05:47 INFO SecurityManager: Changing view acls groups to: 
20/07/06 17:05:47 INFO SecurityManager: Changing modify acls groups to: 
20/07/06 17:05:47 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(yarn, sorokaa); groups with view permissions: Set(); users  with modify permissions: Set(yarn, sorokaa); groups with modify permissions: Set()
20/07/06 17:05:47 DEBUG SecurityManager: Created SSL options for fs: SSLOptions{enabled=false, port=None, keyStore=None, keyStorePassword=None, trustStore=None, trustStorePassword=None, protocol=None, enabledAlgorithms=Set()}
20/07/06 17:05:47 DEBUG TransportServer: Shuffle server started on port: 44552
20/07/06 17:05:47 INFO Utils: Successfully started service 'sparkDriver' on port 44552.
20/07/06 17:05:47 DEBUG SparkEnv: Using serializer: class org.apache.spark.serializer.JavaSerializer
20/07/06 17:05:47 INFO SparkEnv: Registering MapOutputTracker
20/07/06 17:05:47 DEBUG MapOutputTrackerMasterEndpoint: init
20/07/06 17:05:47 INFO SparkEnv: Registering BlockManagerMaster
20/07/06 17:05:47 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
20/07/06 17:05:47 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up
20/07/06 17:05:47 INFO DiskBlockManager: Created local directory at /hadoop/yarn/local/usercache/sorokaa/appcache/application_1594045625213_0020/blockmgr-d7c80f3f-7c34-4680-8d80-549f53cbc522
20/07/06 17:05:47 INFO DiskBlockManager: Created local directory at /var/hadoop/yarn/local/usercache/sorokaa/appcache/application_1594045625213_0020/blockmgr-06c3dce7-50a9-4df0-b2f1-0475f43638b5
20/07/06 17:05:47 DEBUG DiskBlockManager: Adding shutdown hook
20/07/06 17:05:47 INFO MemoryStore: MemoryStore started with capacity 366.3 MB
20/07/06 17:05:47 INFO SparkEnv: Registering OutputCommitCoordinator
20/07/06 17:05:47 DEBUG OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: init
20/07/06 17:05:47 DEBUG SecurityManager: Created SSL options for ui: SSLOptions{enabled=false, port=None, keyStore=None, keyStorePassword=None, trustStore=None, trustStorePassword=None, protocol=None, enabledAlgorithms=Set()}
20/07/06 17:05:47 INFO log: Logging initialized @8513ms
20/07/06 17:05:47 INFO JettyUtils: Adding filter org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter to /jobs, /jobs/json, /jobs/job, /jobs/job/json, /stages, /stages/json, /stages/stage, /stages/stage/json, /stages/pool, /stages/pool/json, /storage, /storage/json, /storage/rdd, /storage/rdd/json, /environment, /environment/json, /executors, /executors/json, /executors/threadDump, /executors/threadDump/json, /static, /, /api, /jobs/job/kill, /stages/stage/kill.
20/07/06 17:05:47 INFO Server: jetty-9.3.z-SNAPSHOT, build timestamp: 2018-06-05T13:11:56-04:00, git hash: 84205aa28f11a4f31f2a3b86d1bba2cc8ab69827
20/07/06 17:05:47 INFO Server: Started @8712ms
20/07/06 17:05:47 INFO AbstractConnector: Started ServerConnector@55e42de5{HTTP/1.1,[http/1.1]}{0.0.0.0:45930}
20/07/06 17:05:47 INFO Utils: Successfully started service 'SparkUI' on port 45930.
20/07/06 17:05:47 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@45e9d484{/jobs,null,AVAILABLE,@Spark}
20/07/06 17:05:47 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@4aee1e70{/jobs/json,null,AVAILABLE,@Spark}
20/07/06 17:05:47 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@49d3424b{/jobs/job,null,AVAILABLE,@Spark}
20/07/06 17:05:47 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@62ee057c{/jobs/job/json,null,AVAILABLE,@Spark}
20/07/06 17:05:47 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@58498acf{/stages,null,AVAILABLE,@Spark}
20/07/06 17:05:47 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@b5b4a97{/stages/json,null,AVAILABLE,@Spark}
20/07/06 17:05:47 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@1e9b98a9{/stages/stage,null,AVAILABLE,@Spark}
20/07/06 17:05:47 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@4919a92a{/stages/stage/json,null,AVAILABLE,@Spark}
20/07/06 17:05:47 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@b053e70{/stages/pool,null,AVAILABLE,@Spark}
20/07/06 17:05:47 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@62ff8f40{/stages/pool/json,null,AVAILABLE,@Spark}
20/07/06 17:05:47 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@40a011a7{/storage,null,AVAILABLE,@Spark}
20/07/06 17:05:47 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@44f4b060{/storage/json,null,AVAILABLE,@Spark}
20/07/06 17:05:47 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@66a0f030{/storage/rdd,null,AVAILABLE,@Spark}
20/07/06 17:05:47 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@78bbe62b{/storage/rdd/json,null,AVAILABLE,@Spark}
20/07/06 17:05:47 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@38d6edb2{/environment,null,AVAILABLE,@Spark}
20/07/06 17:05:47 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@59402d78{/environment/json,null,AVAILABLE,@Spark}
20/07/06 17:05:47 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@38806710{/executors,null,AVAILABLE,@Spark}
20/07/06 17:05:47 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@1fad5495{/executors/json,null,AVAILABLE,@Spark}
20/07/06 17:05:47 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@109981af{/executors/threadDump,null,AVAILABLE,@Spark}
20/07/06 17:05:47 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@486a4529{/executors/threadDump/json,null,AVAILABLE,@Spark}
20/07/06 17:05:47 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@73e903d3{/static,null,AVAILABLE,@Spark}
20/07/06 17:05:47 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@5e317528{/,null,AVAILABLE,@Spark}
20/07/06 17:05:47 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@236d9ce{/api,null,AVAILABLE,@Spark}
20/07/06 17:05:47 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@4df32c67{/jobs/job/kill,null,AVAILABLE,@Spark}
20/07/06 17:05:47 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@7ebeb102{/stages/stage/kill,null,AVAILABLE,@Spark}
20/07/06 17:05:47 INFO SparkUI: Bound SparkUI to 0.0.0.0, and started at http://test25.cm.cluster:45930
20/07/06 17:05:47 INFO YarnClusterScheduler: Created YarnClusterScheduler
20/07/06 17:05:47 INFO SchedulerExtensionServices: Starting Yarn extension services with app application_1594045625213_0020 and attemptId Some(appattempt_1594045625213_0020_000001)
20/07/06 17:05:47 DEBUG TransportServer: Shuffle server started on port: 45586
20/07/06 17:05:47 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 45586.
20/07/06 17:05:47 INFO NettyBlockTransferService: Server created on test25.cm.cluster:45586
20/07/06 17:05:47 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
20/07/06 17:05:47 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, test25.cm.cluster, 45586, None)
20/07/06 17:05:47 DEBUG DefaultTopologyMapper: Got a request for test25.cm.cluster
20/07/06 17:05:47 INFO BlockManagerMasterEndpoint: Registering block manager test25.cm.cluster:45586 with 366.3 MB RAM, BlockManagerId(driver, test25.cm.cluster, 45586, None)
20/07/06 17:05:47 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, test25.cm.cluster, 45586, None)
20/07/06 17:05:47 INFO BlockManager: Initialized BlockManager: BlockManagerId(driver, test25.cm.cluster, 45586, None)
20/07/06 17:05:48 INFO JettyUtils: Adding filter org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter to /metrics/json.
20/07/06 17:05:48 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@6a264198{/metrics/json,null,AVAILABLE,@Spark}
20/07/06 17:05:48 INFO EventLoggingListener: Logging events to hdfs:/spark2-history/application_1594045625213_0020_1
20/07/06 17:05:48 INFO ApplicationProperties: Looking for atlas-application.properties in classpath
20/07/06 17:05:48 INFO ApplicationProperties: Looking for /atlas-application.properties in classpath
20/07/06 17:05:48 INFO ApplicationProperties: Loading atlas-application.properties from null
20/07/06 17:05:48 INFO AbstractConnector: Stopped Spark@55e42de5{HTTP/1.1,[http/1.1]}{0.0.0.0:0}
20/07/06 17:05:48 INFO SparkUI: Stopped Spark web UI at http://test25.cm.cluster:45930
20/07/06 17:05:49 WARN YarnSchedulerBackend$YarnSchedulerEndpoint: Attempted to request executors before the AM has registered!
20/07/06 17:05:49 INFO YarnClusterSchedulerBackend: Shutting down all executors
20/07/06 17:05:49 INFO YarnSchedulerBackend$YarnDriverEndpoint: Asking each executor to shut down
20/07/06 17:05:49 INFO SchedulerExtensionServices: Stopping SchedulerExtensionServices
(serviceOption=None,
 services=List(),
 started=false)
20/07/06 17:05:49 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
20/07/06 17:05:49 INFO MemoryStore: MemoryStore cleared
20/07/06 17:05:49 INFO BlockManager: BlockManager stopped
20/07/06 17:05:49 INFO BlockManagerMaster: BlockManagerMaster stopped
20/07/06 17:05:49 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
20/07/06 17:05:49 INFO SparkContext: Successfully stopped SparkContext
20/07/06 17:05:49 ERROR SparkContext: Error initializing SparkContext.
org.apache.spark.SparkException: Exception when registering SparkListener
        at org.apache.spark.SparkContext.setupAndStartListenerBus(SparkContext.scala:2376)
        at org.apache.spark.SparkContext.<init>(SparkContext.scala:554)
        at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2498)
        at org.apache.spark.SparkContext.getOrCreate(SparkContext.scala)
        at org.apache.livy.rsc.driver.SparkEntries.sc(SparkEntries.java:53)
        at org.apache.livy.rsc.driver.SparkEntries.sparkSession(SparkEntries.java:67)
        at org.apache.livy.repl.AbstractSparkInterpreter.postStart(AbstractSparkInterpreter.scala:69)
        at org.apache.livy.repl.SparkInterpreter$$anonfun$start$1.apply$mcV$sp(SparkInterpreter.scala:88)
        at org.apache.livy.repl.SparkInterpreter$$anonfun$start$1.apply(SparkInterpreter.scala:63)
        at org.apache.livy.repl.SparkInterpreter$$anonfun$start$1.apply(SparkInterpreter.scala:63)
        at org.apache.livy.repl.AbstractSparkInterpreter.restoreContextClassLoader(AbstractSparkInterpreter.scala:340)
        at org.apache.livy.repl.SparkInterpreter.start(SparkInterpreter.scala:63)
        at org.apache.livy.repl.Session$$anonfun$1.apply(Session.scala:128)
        at org.apache.livy.repl.Session$$anonfun$1.apply(Session.scala:122)
        at scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)
        at scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.atlas.AtlasException: Failed to load application properties
        at org.apache.atlas.ApplicationProperties.get(ApplicationProperties.java:143)
        at org.apache.atlas.ApplicationProperties.get(ApplicationProperties.java:95)
        at com.hortonworks.spark.atlas.AtlasClientConf.configuration$lzycompute(AtlasClientConf.scala:25)
        at com.hortonworks.spark.atlas.AtlasClientConf.configuration(AtlasClientConf.scala:25)
        at com.hortonworks.spark.atlas.AtlasClientConf.get(AtlasClientConf.scala:50)
        at com.hortonworks.spark.atlas.AtlasClient$.atlasClient(AtlasClient.scala:88)
        at com.hortonworks.spark.atlas.SparkAtlasEventTracker.<init>(SparkAtlasEventTracker.scala:39)
        at com.hortonworks.spark.atlas.SparkAtlasEventTracker.<init>(SparkAtlasEventTracker.scala:43)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at org.apache.spark.util.Utils$$anonfun$loadExtensions$1.apply(Utils.scala:2747)
        at org.apache.spark.util.Utils$$anonfun$loadExtensions$1.apply(Utils.scala:2736)
        at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
        at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
        at scala.collection.mutable.ArraySeq.foreach(ArraySeq.scala:74)
        at scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:241)
        at scala.collection.AbstractTraversable.flatMap(Traversable.scala:104)
        at org.apache.spark.util.Utils$.loadExtensions(Utils.scala:2736)
        at org.apache.spark.SparkContext$$anonfun$setupAndStartListenerBus$1.apply(SparkContext.scala:2365)
        at org.apache.spark.SparkContext$$anonfun$setupAndStartListenerBus$1.apply(SparkContext.scala:2364)
        at scala.Option.foreach(Option.scala:257)
        at org.apache.spark.SparkContext.setupAndStartListenerBus(SparkContext.scala:2364)
        ... 18 more
Caused by: org.apache.commons.configuration.ConfigurationException: Cannot locate configuration source null
        at org.apache.commons.configuration.AbstractFileConfiguration.load(AbstractFileConfiguration.java:259)
        at org.apache.commons.configuration.AbstractFileConfiguration.load(AbstractFileConfiguration.java:238)
        at org.apache.commons.configuration.AbstractFileConfiguration.<init>(AbstractFileConfiguration.java:197)
        at org.apache.commons.configuration.PropertiesConfiguration.<init>(PropertiesConfiguration.java:284)
        at org.apache.atlas.ApplicationProperties.<init>(ApplicationProperties.java:78)
        at org.apache.atlas.ApplicationProperties.get(ApplicationProperties.java:134)
        ... 41 more
20/07/06 17:05:49 INFO SparkContext: SparkContext already stopped.
20/07/06 17:05:49 ERROR ApplicationMaster: User class threw exception: org.apache.spark.SparkException: Exception when registering SparkListener
org.apache.spark.SparkException: Exception when registering SparkListener
        at org.apache.spark.SparkContext.setupAndStartListenerBus(SparkContext.scala:2376)
        at org.apache.spark.SparkContext.<init>(SparkContext.scala:554)
        at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2498)
        at org.apache.spark.SparkContext.getOrCreate(SparkContext.scala)
        at org.apache.livy.rsc.driver.SparkEntries.sc(SparkEntries.java:53)
        at org.apache.livy.rsc.driver.SparkEntries.sparkSession(SparkEntries.java:67)
        at org.apache.livy.repl.AbstractSparkInterpreter.postStart(AbstractSparkInterpreter.scala:69)
        at org.apache.livy.repl.SparkInterpreter$$anonfun$start$1.apply$mcV$sp(SparkInterpreter.scala:88)
        at org.apache.livy.repl.SparkInterpreter$$anonfun$start$1.apply(SparkInterpreter.scala:63)
        at org.apache.livy.repl.SparkInterpreter$$anonfun$start$1.apply(SparkInterpreter.scala:63)
        at org.apache.livy.repl.AbstractSparkInterpreter.restoreContextClassLoader(AbstractSparkInterpreter.scala:340)
        at org.apache.livy.repl.SparkInterpreter.start(SparkInterpreter.scala:63)
        at org.apache.livy.repl.Session$$anonfun$1.apply(Session.scala:128)
        at org.apache.livy.repl.Session$$anonfun$1.apply(Session.scala:122)
        at scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)
        at scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.atlas.AtlasException: Failed to load application properties
        at org.apache.atlas.ApplicationProperties.get(ApplicationProperties.java:143)
        at org.apache.atlas.ApplicationProperties.get(ApplicationProperties.java:95)
        at com.hortonworks.spark.atlas.AtlasClientConf.configuration$lzycompute(AtlasClientConf.scala:25)
        at com.hortonworks.spark.atlas.AtlasClientConf.configuration(AtlasClientConf.scala:25)
        at com.hortonworks.spark.atlas.AtlasClientConf.get(AtlasClientConf.scala:50)
        at com.hortonworks.spark.atlas.AtlasClient$.atlasClient(AtlasClient.scala:88)
        at com.hortonworks.spark.atlas.SparkAtlasEventTracker.<init>(SparkAtlasEventTracker.scala:39)
        at com.hortonworks.spark.atlas.SparkAtlasEventTracker.<init>(SparkAtlasEventTracker.scala:43)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at org.apache.spark.util.Utils$$anonfun$loadExtensions$1.apply(Utils.scala:2747)
        at org.apache.spark.util.Utils$$anonfun$loadExtensions$1.apply(Utils.scala:2736)
        at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
        at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
        at scala.collection.mutable.ArraySeq.foreach(ArraySeq.scala:74)
        at scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:241)
        at scala.collection.AbstractTraversable.flatMap(Traversable.scala:104)
        at org.apache.spark.util.Utils$.loadExtensions(Utils.scala:2736)
        at org.apache.spark.SparkContext$$anonfun$setupAndStartListenerBus$1.apply(SparkContext.scala:2365)
        at org.apache.spark.SparkContext$$anonfun$setupAndStartListenerBus$1.apply(SparkContext.scala:2364)
        at scala.Option.foreach(Option.scala:257)
        at org.apache.spark.SparkContext.setupAndStartListenerBus(SparkContext.scala:2364)
        ... 18 more
Caused by: org.apache.commons.configuration.ConfigurationException: Cannot locate configuration source null
        at org.apache.commons.configuration.AbstractFileConfiguration.load(AbstractFileConfiguration.java:259)
        at org.apache.commons.configuration.AbstractFileConfiguration.load(AbstractFileConfiguration.java:238)
        at org.apache.commons.configuration.AbstractFileConfiguration.<init>(AbstractFileConfiguration.java:197)
        at org.apache.commons.configuration.PropertiesConfiguration.<init>(PropertiesConfiguration.java:284)
        at org.apache.atlas.ApplicationProperties.<init>(ApplicationProperties.java:78)
        at org.apache.atlas.ApplicationProperties.get(ApplicationProperties.java:134)
        ... 41 more
20/07/06 17:05:49 INFO ApplicationMaster: Final app status: FAILED, exitCode: 13, (reason: User class threw exception: org.apache.spark.SparkException: Exception when registering SparkListener
        at org.apache.spark.SparkContext.setupAndStartListenerBus(SparkContext.scala:2376)
        at org.apache.spark.SparkContext.<init>(SparkContext.scala:554)
        at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2498)
        at org.apache.spark.SparkContext.getOrCreate(SparkContext.scala)
        at org.apache.livy.rsc.driver.SparkEntries.sc(SparkEntries.java:53)
        at org.apache.livy.rsc.driver.SparkEntries.sparkSession(SparkEntries.java:67)
        at org.apache.livy.repl.AbstractSparkInterpreter.postStart(AbstractSparkInterpreter.scala:69)
        at org.apache.livy.repl.SparkInterpreter$$anonfun$start$1.apply$mcV$sp(SparkInterpreter.scala:88)
        at org.apache.livy.repl.SparkInterpreter$$anonfun$start$1.apply(SparkInterpreter.scala:63)
        at org.apache.livy.repl.SparkInterpreter$$anonfun$start$1.apply(SparkInterpreter.scala:63)
        at org.apache.livy.repl.AbstractSparkInterpreter.restoreContextClassLoader(AbstractSparkInterpreter.scala:340)
        at org.apache.livy.repl.SparkInterpreter.start(SparkInterpreter.scala:63)
        at org.apache.livy.repl.Session$$anonfun$1.apply(Session.scala:128)
        at org.apache.livy.repl.Session$$anonfun$1.apply(Session.scala:122)
        at scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)
        at scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.atlas.AtlasException: Failed to load application properties
        at org.apache.atlas.ApplicationProperties.get(ApplicationProperties.java:143)
        at org.apache.atlas.ApplicationProperties.get(ApplicationProperties.java:95)
        at com.hortonworks.spark.atlas.AtlasClientConf.configuration$lzycompute(AtlasClientConf.scala:25)
        at com.hortonworks.spark.atlas.AtlasClientConf.configuration(AtlasClientConf.scala:25)
        at com.hortonworks.spark.atlas.AtlasClientConf.get(AtlasClientConf.scala:50)
        at com.hortonworks.spark.atlas.AtlasClient$.atlasClient(AtlasClient.scala:88)
        at com.hortonworks.spark.atlas.SparkAtlasEventTracker.<init>(SparkAtlasEventTracker.scala:39)
        at com.hortonworks.spark.atlas.SparkAtlasEventTracker.<init>(SparkAtlasEventTracker.scala:43)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at org.apache.spark.util.Utils$$anonfun$loadExtensions$1.apply(Utils.scala:2747)
        at org.apache.spark.util.Utils$$anonfun$loadExtensions$1.apply(Utils.scala:2736)
        at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
        at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
        at scala.collection.mutable.ArraySeq.foreach(ArraySeq.scala:74)
        at scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:241)
        at scala.collection.AbstractTraversable.flatMap(Traversable.scala:104)
        at org.apache.spark.util.Utils$.loadExtensions(Utils.scala:2736)
        at org.apache.spark.SparkContext$$anonfun$setupAndStartListenerBus$1.apply(SparkContext.scala:2365)
        at org.apache.spark.SparkContext$$anonfun$setupAndStartListenerBus$1.apply(SparkContext.scala:2364)
        at scala.Option.foreach(Option.scala:257)
        at org.apache.spark.SparkContext.setupAndStartListenerBus(SparkContext.scala:2364)
        ... 18 more
Caused by: org.apache.commons.configuration.ConfigurationException: Cannot locate configuration source null
        at org.apache.commons.configuration.AbstractFileConfiguration.load(AbstractFileConfiguration.java:259)
        at org.apache.commons.configuration.AbstractFileConfiguration.load(AbstractFileConfiguration.java:238)
        at org.apache.commons.configuration.AbstractFileConfiguration.<init>(AbstractFileConfiguration.java:197)
        at org.apache.commons.configuration.PropertiesConfiguration.<init>(PropertiesConfiguration.java:284)
        at org.apache.atlas.ApplicationProperties.<init>(ApplicationProperties.java:78)
        at org.apache.atlas.ApplicationProperties.get(ApplicationProperties.java:134)
        ... 41 more
)
20/07/06 17:05:49 ERROR ApplicationMaster: Uncaught exception: 
org.apache.spark.SparkException: Exception thrown in awaitResult: 
        at org.apache.spark.util.ThreadUtils$.awaitResult(ThreadUtils.scala:205)
        at org.apache.spark.deploy.yarn.ApplicationMaster.runDriver(ApplicationMaster.scala:498)
        at org.apache.spark.deploy.yarn.ApplicationMaster.org$apache$spark$deploy$yarn$ApplicationMaster$$runImpl(ApplicationMaster.scala:345)
        at org.apache.spark.deploy.yarn.ApplicationMaster$$anonfun$run$2.apply$mcV$sp(ApplicationMaster.scala:260)
        at org.apache.spark.deploy.yarn.ApplicationMaster$$anonfun$run$2.apply(ApplicationMaster.scala:260)
        at org.apache.spark.deploy.yarn.ApplicationMaster$$anonfun$run$2.apply(ApplicationMaster.scala:260)
        at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$5.run(ApplicationMaster.scala:815)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
        at org.apache.spark.deploy.yarn.ApplicationMaster.doAsUser(ApplicationMaster.scala:814)
        at org.apache.spark.deploy.yarn.ApplicationMaster.run(ApplicationMaster.scala:259)
        at org.apache.spark.deploy.yarn.ApplicationMaster$.main(ApplicationMaster.scala:839)
        at org.apache.spark.deploy.yarn.ApplicationMaster.main(ApplicationMaster.scala)
Caused by: org.apache.spark.SparkException: Exception when registering SparkListener
        at org.apache.spark.SparkContext.setupAndStartListenerBus(SparkContext.scala:2376)
        at org.apache.spark.SparkContext.<init>(SparkContext.scala:554)
        at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2498)
        at org.apache.spark.SparkContext.getOrCreate(SparkContext.scala)
        at org.apache.livy.rsc.driver.SparkEntries.sc(SparkEntries.java:53)
        at org.apache.livy.rsc.driver.SparkEntries.sparkSession(SparkEntries.java:67)
        at org.apache.livy.repl.AbstractSparkInterpreter.postStart(AbstractSparkInterpreter.scala:69)
        at org.apache.livy.repl.SparkInterpreter$$anonfun$start$1.apply$mcV$sp(SparkInterpreter.scala:88)
        at org.apache.livy.repl.SparkInterpreter$$anonfun$start$1.apply(SparkInterpreter.scala:63)
        at org.apache.livy.repl.SparkInterpreter$$anonfun$start$1.apply(SparkInterpreter.scala:63)
        at org.apache.livy.repl.AbstractSparkInterpreter.restoreContextClassLoader(AbstractSparkInterpreter.scala:340)
        at org.apache.livy.repl.SparkInterpreter.start(SparkInterpreter.scala:63)
        at org.apache.livy.repl.Session$$anonfun$1.apply(Session.scala:128)
        at org.apache.livy.repl.Session$$anonfun$1.apply(Session.scala:122)
        at scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)
        at scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.atlas.AtlasException: Failed to load application properties
        at org.apache.atlas.ApplicationProperties.get(ApplicationProperties.java:143)
        at org.apache.atlas.ApplicationProperties.get(ApplicationProperties.java:95)
        at com.hortonworks.spark.atlas.AtlasClientConf.configuration$lzycompute(AtlasClientConf.scala:25)
        at com.hortonworks.spark.atlas.AtlasClientConf.configuration(AtlasClientConf.scala:25)
        at com.hortonworks.spark.atlas.AtlasClientConf.get(AtlasClientConf.scala:50)
        at com.hortonworks.spark.atlas.AtlasClient$.atlasClient(AtlasClient.scala:88)
        at com.hortonworks.spark.atlas.SparkAtlasEventTracker.<init>(SparkAtlasEventTracker.scala:39)
        at com.hortonworks.spark.atlas.SparkAtlasEventTracker.<init>(SparkAtlasEventTracker.scala:43)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at org.apache.spark.util.Utils$$anonfun$loadExtensions$1.apply(Utils.scala:2747)
        at org.apache.spark.util.Utils$$anonfun$loadExtensions$1.apply(Utils.scala:2736)
        at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
        at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
        at scala.collection.mutable.ArraySeq.foreach(ArraySeq.scala:74)
        at scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:241)
        at scala.collection.AbstractTraversable.flatMap(Traversable.scala:104)
        at org.apache.spark.util.Utils$.loadExtensions(Utils.scala:2736)
        at org.apache.spark.SparkContext$$anonfun$setupAndStartListenerBus$1.apply(SparkContext.scala:2365)
        at org.apache.spark.SparkContext$$anonfun$setupAndStartListenerBus$1.apply(SparkContext.scala:2364)
        at scala.Option.foreach(Option.scala:257)
        at org.apache.spark.SparkContext.setupAndStartListenerBus(SparkContext.scala:2364)
        ... 18 more
Caused by: org.apache.commons.configuration.ConfigurationException: Cannot locate configuration source null
        at org.apache.commons.configuration.AbstractFileConfiguration.load(AbstractFileConfiguration.java:259)
        at org.apache.commons.configuration.AbstractFileConfiguration.load(AbstractFileConfiguration.java:238)
        at org.apache.commons.configuration.AbstractFileConfiguration.<init>(AbstractFileConfiguration.java:197)
        at org.apache.commons.configuration.PropertiesConfiguration.<init>(PropertiesConfiguration.java:284)
        at org.apache.atlas.ApplicationProperties.<init>(ApplicationProperties.java:78)
        at org.apache.atlas.ApplicationProperties.get(ApplicationProperties.java:134)
        ... 41 more
20/07/06 17:05:49 INFO ApplicationMaster: Deleting staging directory hdfs://test05.cm.cluster:8020/user/sorokaa/.sparkStaging/application_1594045625213_0020
20/07/06 17:05:49 INFO ShutdownHookManager: Shutdown hook called
20/07/06 17:05:49 INFO ShutdownHookManager: Deleting directory /hadoop/yarn/local/usercache/sorokaa/appcache/application_1594045625213_0020/spark-cfb471fb-6b99-455d-b57f-cd7054db2135
20/07/06 17:05:49 INFO ShutdownHookManager: Deleting directory /var/hadoop/yarn/local/usercache/sorokaa/appcache/application_1594045625213_0020/spark-1c359db0-b7bd-4166-b294-9612bc3d547b
20/07/06 17:05:49 ERROR ApplicationMaster: RECEIVED SIGNAL TERM
ajs6f commented 3 years ago

For anyone who happens on this issue in the future, I was able to work around this by adding an entry to spark.driver.extraClassPath.