search

citiususc / SparkBWA

SparkBWA is a new tool that exploits the capabilities of a Big Data technology as Apache Spark to boost the performance of one of the most widely adopted sequence aligner, the Burrows-Wheeler Aligner (BWA).
GNU General Public License v3.0
69 stars 26 forks source link

ERROR LiveListenerBus: Listener EventLoggingListener threw an exception #16

Closed wangzhenkeep closed 8 years ago

wangzhenkeep commented 8 years ago

I got something wrong.Anyone can help me。 Thanks！

my command： nohup spark-submit --class SparkBWA --master yarn-client --driver-memory 6000m --executor-memory 6000m --executor-cores 1 --archives /share/data/wangzhen/project/2016/spark_alignment/SparkBWA/build/bwa.zip --verbose --num-executors 1 /share/data/wangzhen/project/2016/spark_alignment/SparkBWA/build/SparkBWA.jar -algorithm mem -reads paired -index /share/data/wangzhen/project/2016/spark_alignment/SparkBWA/test/database/16s_refseq.fna.formatted -partitions 1 /wangzhen/sparkBWA/10_1.fq /wangzhen/sparkBWA/10_2.fq /wangzhen/sparkBWA/result

The error： Using properties file: /usr/lib/spark151/conf/spark-defaults.conf Adding default property: spark.port.maxRetries=40 Adding default property: spark.akka.timeout=300 Adding default property: spark.serializer=org.apache.spark.serializer.KryoSerializer Adding default property: spark.executor.extraJavaOptions=-Djava.library.path=/share/data/wangzhen/project/2016/spark_alignment/SparkBWA/build/bwa.zip Adding default property: spark.eventLog.enabled=true Adding default property: spark.driver.maxResultSize=6g Adding default property: spark.worker.cleanup.appDataTtl=24*3600 Adding default property: spark.network.timeout=300 Adding default property: spark.storage.memoryFraction=0.8 Adding default property: spark.driver.memory=8g Adding default property: spark.default.parallelism=48 Adding default property: spark.shuffle.spill=true Adding default property: spark.master=spark://master:7077 Adding default property: spark.shuffle.file.buffer=96k Adding default property: spark.local.dir=/data/sparkTmp Adding default property: spark.eventLog.dir=hdfs://master:9000/sparkHistoryLog Adding default property: spark.worker.timeout=120 Adding default property: spark.eventLog.compress=true Adding default property: spark.task.cpus=1 Adding default property: spark.shuffle.consolidateFiles=true Adding default property: spark.task.maxFailures=8 Parsed arguments: master yarn-client deployMode null executorMemory 6000m executorCores 1 totalExecutorCores null propertiesFile /usr/lib/spark151/conf/spark-defaults.conf driverMemory 6000m driverCores null driverExtraClassPath null driverExtraLibraryPath null driverExtraJavaOptions null supervise false queue null numExecutors 1 files null pyFiles null archives file:/share/data/wangzhen/project/2016/spark_alignment/SparkBWA/build/bwa.zip mainClass SparkBWA primaryResource file:/share/data/wangzhen/project/2016/spark_alignment/SparkBWA/build/SparkBWA.jar name SparkBWA childArgs [-algorithm mem -reads paired -index /share/data/wangzhen/project/2016/spark_alignment/SparkBWA/test/database/16s_refseq.fna.formatted -partitions 1 /wangzhen/sparkBWA/10_1.fq /wangzhen/sparkBWA/10_2.fq /wangzhen/sparkBWA/result] jars null packages null packagesExclusions null repositories null verbose true

Spark properties used, including those specified through --conf and those from the properties file /usr/lib/spark151/conf/spark-defaults.conf: spark.local.dir -> /data/sparkTmp spark.default.parallelism -> 48 spark.driver.memory -> 6000m spark.network.timeout -> 300 spark.worker.cleanup.appDataTtl -> 24*3600 spark.eventLog.compress -> true spark.worker.timeout -> 120 spark.eventLog.enabled -> true spark.akka.timeout -> 300 spark.shuffle.consolidateFiles -> true spark.serializer -> org.apache.spark.serializer.KryoSerializer spark.task.cpus -> 1 spark.executor.extraJavaOptions -> -Djava.library.path=/share/data/wangzhen/project/2016/spark_alignment/SparkBWA/build/bwa.zip spark.shuffle.spill -> true spark.task.maxFailures -> 8 spark.eventLog.dir -> hdfs://master:9000/sparkHistoryLog spark.master -> spark://master:7077 spark.driver.maxResultSize -> 6g spark.port.maxRetries -> 40 spark.storage.memoryFraction -> 0.8 spark.shuffle.file.buffer -> 96k

Main class: SparkBWA Arguments: -algorithm mem -reads paired -index /share/data/wangzhen/project/2016/spark_alignment/SparkBWA/test/database/16s_refseq.fna.formatted -partitions 1 /wangzhen/sparkBWA/10_1.fq /wangzhen/sparkBWA/10_2.fq /wangzhen/sparkBWA/result System properties: spark.local.dir -> /data/sparkTmp spark.default.parallelism -> 48 spark.driver.memory -> 6000m spark.network.timeout -> 300 spark.executor.memory -> 6000m spark.executor.instances -> 1 spark.worker.cleanup.appDataTtl -> 24*3600 spark.eventLog.compress -> true spark.worker.timeout -> 120 spark.eventLog.enabled -> true SPARK_SUBMIT -> true spark.akka.timeout -> 300 spark.shuffle.consolidateFiles -> true spark.serializer -> org.apache.spark.serializer.KryoSerializer spark.task.cpus -> 1 spark.executor.extraJavaOptions -> -Djava.library.path=/share/data/wangzhen/project/2016/spark_alignment/SparkBWA/build/bwa.zip spark.app.name -> SparkBWA spark.shuffle.spill -> true spark.jars -> file:/share/data/wangzhen/project/2016/spark_alignment/SparkBWA/build/SparkBWA.jar spark.task.maxFailures -> 8 spark.yarn.dist.archives -> file:/share/data/wangzhen/project/2016/spark_alignment/SparkBWA/build/bwa.zip spark.submit.deployMode -> client spark.eventLog.dir -> hdfs://master:9000/sparkHistoryLog spark.driver.maxResultSize -> 6g spark.master -> yarn-client spark.port.maxRetries -> 40 spark.executor.cores -> 1 spark.shuffle.file.buffer -> 96k spark.storage.memoryFraction -> 0.8 Classpath elements: file:/share/data/wangzhen/project/2016/spark_alignment/SparkBWA/build/SparkBWA.jar

16/07/13 10:54:39 INFO BwaOptions: JMAbuin:: Received argument: -algorithm 16/07/13 10:54:39 INFO BwaOptions: JMAbuin:: Received argument: mem 16/07/13 10:54:39 INFO BwaOptions: JMAbuin:: Received argument: -reads 16/07/13 10:54:39 INFO BwaOptions: JMAbuin:: Received argument: paired 16/07/13 10:54:39 INFO BwaOptions: JMAbuin:: Received argument: -index 16/07/13 10:54:39 INFO BwaOptions: JMAbuin:: Received argument: /share/data/wangzhen/project/2016/spark_alignment/SparkBWA/test/database/16s_refseq.fna.formatted 16/07/13 10:54:39 INFO BwaOptions: JMAbuin:: Received argument: -partitions 16/07/13 10:54:39 INFO BwaOptions: JMAbuin:: Received argument: 1 16/07/13 10:54:39 INFO BwaOptions: JMAbuin:: Received argument: /wangzhen/sparkBWA/10_1.fq 16/07/13 10:54:39 INFO BwaOptions: JMAbuin:: Received argument: /wangzhen/sparkBWA/10_2.fq 16/07/13 10:54:39 INFO BwaOptions: JMAbuin:: Received argument: /wangzhen/sparkBWA/result 16/07/13 10:54:39 INFO SparkContext: Running Spark version 1.5.1 16/07/13 10:54:39 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 16/07/13 10:54:39 WARN SparkConf: In Spark 1.0 and later spark.local.dir will be overridden by the value set by the cluster manager (via SPARK_LOCAL_DIRS in mesos/standalone and LOCAL_DIRS in YARN). 16/07/13 10:54:39 INFO SecurityManager: Changing view acls to: root 16/07/13 10:54:39 INFO SecurityManager: Changing modify acls to: root 16/07/13 10:54:39 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root); users with modify permissions: Set(root) 16/07/13 10:54:40 INFO Slf4jLogger: Slf4jLogger started 16/07/13 10:54:40 INFO Remoting: Starting remoting 16/07/13 10:54:40 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriver@192.168.100.17:33949] 16/07/13 10:54:40 INFO Utils: Successfully started service 'sparkDriver' on port 33949. 16/07/13 10:54:40 INFO SparkEnv: Registering MapOutputTracker 16/07/13 10:54:40 INFO SparkEnv: Registering BlockManagerMaster 16/07/13 10:54:41 INFO DiskBlockManager: Created local directory at /data/sparkTmp/blockmgr-18818f3c-9076-43fa-affa-4018282f974b 16/07/13 10:54:41 INFO MemoryStore: MemoryStore started with capacity 4.0 GB 16/07/13 10:54:41 INFO HttpFileServer: HTTP File server directory is /data/sparkTmp/spark-0eb513a2-e2e1-4c8d-b1c2-044d7211cfc0/httpd-61146258-69d4-4c7f-9dda-337ec907943b 16/07/13 10:54:41 INFO HttpServer: Starting HTTP Server 16/07/13 10:54:41 INFO Utils: Successfully started service 'HTTP file server' on port 40187. 16/07/13 10:54:41 INFO SparkEnv: Registering OutputCommitCoordinator 16/07/13 10:54:41 INFO Utils: Successfully started service 'SparkUI' on port 4040. 16/07/13 10:54:41 INFO SparkUI: Started SparkUI at http://192.168.100.17:4040 16/07/13 10:54:41 INFO SparkContext: Added JAR file:/share/data/wangzhen/project/2016/spark_alignment/SparkBWA/build/SparkBWA.jar at http://192.168.100.17:40187/jars/SparkBWA.jar with timestamp 1468378481570 16/07/13 10:54:41 WARN MetricsSystem: Using default name DAGScheduler for source because spark.app.id is not set. 16/07/13 10:54:41 INFO RMProxy: Connecting to ResourceManager at master/192.168.100.17:8032 16/07/13 10:54:41 INFO Client: Requesting a new application from cluster with 4 NodeManagers 16/07/13 10:54:42 INFO Client: Verifying our application has not requested more than the maximum memory capability of the cluster (8192 MB per container) 16/07/13 10:54:42 INFO Client: Will allocate AM container, with 896 MB memory including 384 MB overhead 16/07/13 10:54:42 INFO Client: Setting up container launch context for our AM 16/07/13 10:54:42 INFO Client: Setting up the launch environment for our AM container 16/07/13 10:54:42 INFO Client: Preparing resources for our AM container 16/07/13 10:54:42 INFO Client: Uploading resource file:/usr/lib/spark151/lib/spark-assembly-1.5.1-hadoop2.6.0.jar -> hdfs://master:9000/user/root/.sparkStaging/application_1468374952451_0011/spark-assembly-1.5.1-hadoop2.6.0.jar 16/07/13 10:54:50 INFO Client: Uploading resource file:/share/data/wangzhen/project/2016/spark_alignment/SparkBWA/build/bwa.zip -> hdfs://master:9000/user/root/.sparkStaging/application_1468374952451_0011/bwa.zip 16/07/13 10:54:50 INFO Client: Uploading resource file:/data/sparkTmp/spark-0eb513a2-e2e1-4c8d-b1c2-044d7211cfc0/spark_conf3121139071193347484.zip -> hdfs://master:9000/user/root/.sparkStaging/application_1468374952451_0011/spark_conf3121139071193347484.zip 16/07/13 10:54:50 INFO SecurityManager: Changing view acls to: root 16/07/13 10:54:50 INFO SecurityManager: Changing modify acls to: root 16/07/13 10:54:50 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root); users with modify permissions: Set(root) 16/07/13 10:54:50 INFO Client: Submitting application 11 to ResourceManager 16/07/13 10:54:50 INFO YarnClientImpl: Submitted application application_1468374952451_0011 16/07/13 10:54:51 INFO Client: Application report for application_1468374952451_0011 (state: ACCEPTED) 16/07/13 10:54:51 INFO Client: client token: N/A diagnostics: N/A ApplicationMaster host: N/A ApplicationMaster RPC port: -1 queue: default start time: 1468378490370 final status: UNDEFINED tracking URL: http://master:8089/proxy/application_1468374952451_0011/ user: root 16/07/13 10:54:52 INFO Client: Application report for application_1468374952451_0011 (state: ACCEPTED) 16/07/13 10:54:53 INFO Client: Application report for application_1468374952451_0011 (state: ACCEPTED) 16/07/13 10:54:54 INFO Client: Application report for application_1468374952451_0011 (state: ACCEPTED) 16/07/13 10:54:55 INFO Client: Application report for application_1468374952451_0011 (state: ACCEPTED) 16/07/13 10:54:56 INFO Client: Application report for application_1468374952451_0011 (state: ACCEPTED) 16/07/13 10:54:57 INFO YarnSchedulerBackend$YarnSchedulerEndpoint: ApplicationMaster registered as AkkaRpcEndpointRef(Actor[akka.tcp://sparkYarnAM@192.168.100.22:40327/user/YarnAM#-347527355]) 16/07/13 10:54:57 INFO YarnClientSchedulerBackend: Add WebUI Filter. org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter, Map(PROXY_HOSTS -> master, PROXY_URI_BASES -> http://master:8089/proxy/application_1468374952451_0011), /proxy/application_1468374952451_0011 16/07/13 10:54:57 INFO JettyUtils: Adding filter: org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter 16/07/13 10:54:57 INFO Client: Application report for application_1468374952451_0011 (state: RUNNING) 16/07/13 10:54:57 INFO Client: client token: N/A diagnostics: N/A ApplicationMaster host: 192.168.100.22 ApplicationMaster RPC port: 0 queue: default start time: 1468378490370 final status: UNDEFINED tracking URL: http://master:8089/proxy/application_1468374952451_0011/ user: root 16/07/13 10:54:57 INFO YarnClientSchedulerBackend: Application application_1468374952451_0011 has started running. 16/07/13 10:54:57 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 42843. 16/07/13 10:54:57 INFO NettyBlockTransferService: Server created on 42843 16/07/13 10:54:57 INFO BlockManagerMaster: Trying to register BlockManager 16/07/13 10:54:57 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.100.17:42843 with 4.0 GB RAM, BlockManagerId(driver, 192.168.100.17, 42843) 16/07/13 10:54:57 INFO BlockManagerMaster: Registered BlockManager 16/07/13 10:54:57 INFO EventLoggingListener: Logging events to hdfs://master:9000/sparkHistoryLog/application_1468374952451_0011.snappy 16/07/13 10:55:07 INFO YarnClientSchedulerBackend: Registered executor: AkkaRpcEndpointRef(Actor[akka.tcp://sparkExecutor@slave2:37212/user/Executor#574201963]) with ID 1 16/07/13 10:55:07 INFO YarnClientSchedulerBackend: SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.8 16/07/13 10:55:07 INFO BwaInterpreter: JMAbuin:: Starting BWA 16/07/13 10:55:07 INFO BwaInterpreter: JMAbuin::Not sorting in HDFS. Timing: 4690973152386 16/07/13 10:55:07 INFO BlockManagerMasterEndpoint: Registering block manager slave2:46841 with 4.0 GB RAM, BlockManagerId(1, slave2, 46841) 16/07/13 10:55:07 ERROR LiveListenerBus: Listener EventLoggingListener threw an exception java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.spark.scheduler.EventLoggingListener$$anonfun$logEvent$3.apply(EventLoggingListener.scala:148) at org.apache.spark.scheduler.EventLoggingListener$$anonfun$logEvent$3.apply(EventLoggingListener.scala:148) at scala.Option.foreach(Option.scala:236) at org.apache.spark.scheduler.EventLoggingListener.logEvent(EventLoggingListener.scala:148) at org.apache.spark.scheduler.EventLoggingListener.onBlockManagerAdded(EventLoggingListener.scala:176) at org.apache.spark.scheduler.SparkListenerBus$class.onPostEvent(SparkListenerBus.scala:46) at org.apache.spark.scheduler.LiveListenerBus.onPostEvent(LiveListenerBus.scala:31) at org.apache.spark.scheduler.LiveListenerBus.onPostEvent(LiveListenerBus.scala:31) at org.apache.spark.util.ListenerBus$class.postToAll(ListenerBus.scala:56) at org.apache.spark.util.AsynchronousListenerBus.postToAll(AsynchronousListenerBus.scala:37) at org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1.apply$mcV$sp(AsynchronousListenerBus.scala:79) at org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1136) at org.apache.spark.util.AsynchronousListenerBus$$anon$1.run(AsynchronousListenerBus.scala:63) Caused by: java.io.IOException: Filesystem closed at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:795) at org.apache.hadoop.hdfs.DFSOutputStream.flushOrSync(DFSOutputStream.java:1985) at org.apache.hadoop.hdfs.DFSOutputStream.hflush(DFSOutputStream.java:1946) at org.apache.hadoop.fs.FSDataOutputStream.hflush(FSDataOutputStream.java:130) ... 17 more 16/07/13 10:55:08 INFO MemoryStore: ensureFreeSpace(234648) called with curMem=0, maxMem=4341104640 16/07/13 10:55:08 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 229.1 KB, free 4.0 GB) 16/07/13 10:55:08 INFO MemoryStore: ensureFreeSpace(20248) called with curMem=234648, maxMem=4341104640 16/07/13 10:55:08 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 19.8 KB, free 4.0 GB) 16/07/13 10:55:08 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on 192.168.100.17:42843 (size: 19.8 KB, free: 4.0 GB) 16/07/13 10:55:08 INFO SparkContext: Created broadcast 0 from newAPIHadoopFile at BwaInterpreter.java:246 16/07/13 10:55:08 INFO MemoryStore: ensureFreeSpace(234648) called with curMem=254896, maxMem=4341104640 16/07/13 10:55:08 INFO MemoryStore: Block broadcast_1 stored as values in memory (estimated size 229.1 KB, free 4.0 GB) 16/07/13 10:55:08 INFO MemoryStore: ensureFreeSpace(20248) called with curMem=489544, maxMem=4341104640 16/07/13 10:55:08 INFO MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 19.8 KB, free 4.0 GB) 16/07/13 10:55:08 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on 192.168.100.17:42843 (size: 19.8 KB, free: 4.0 GB) 16/07/13 10:55:08 INFO SparkContext: Created broadcast 1 from newAPIHadoopFile at BwaInterpreter.java:247 16/07/13 10:55:08 INFO FileInputFormat: Total input paths to process : 1 16/07/13 10:55:08 INFO FileInputFormat: Total input paths to process : 1 16/07/13 10:55:08 INFO NewHadoopRDD: Removing RDD 0 from persistence list 16/07/13 10:55:08 INFO BlockManager: Removing RDD 0 16/07/13 10:55:08 INFO NewHadoopRDD: Removing RDD 1 from persistence list 16/07/13 10:55:08 INFO BlockManager: Removing RDD 1 16/07/13 10:55:08 ERROR LiveListenerBus: Listener EventLoggingListener threw an exception java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.spark.scheduler.EventLoggingListener$$anonfun$logEvent$3.apply(EventLoggingListener.scala:148) at org.apache.spark.scheduler.EventLoggingListener$$anonfun$logEvent$3.apply(EventLoggingListener.scala:148) at scala.Option.foreach(Option.scala:236) at org.apache.spark.scheduler.EventLoggingListener.logEvent(EventLoggingListener.scala:148) at org.apache.spark.scheduler.EventLoggingListener.onUnpersistRDD(EventLoggingListener.scala:184) at org.apache.spark.scheduler.SparkListenerBus$class.onPostEvent(SparkListenerBus.scala:50) at org.apache.spark.scheduler.LiveListenerBus.onPostEvent(LiveListenerBus.scala:31) at org.apache.spark.scheduler.LiveListenerBus.onPostEvent(LiveListenerBus.scala:31) at org.apache.spark.util.ListenerBus$class.postToAll(ListenerBus.scala:56) at org.apache.spark.util.AsynchronousListenerBus.postToAll(AsynchronousListenerBus.scala:37) at org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1.apply$mcV$sp(AsynchronousListenerBus.scala:79) at org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1136) at org.apache.spark.util.AsynchronousListenerBus$$anon$1.run(AsynchronousListenerBus.scala:63) Caused by: java.io.IOException: Filesystem closed at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:795) at org.apache.hadoop.hdfs.DFSOutputStream.flushOrSync(DFSOutputStream.java:1985) at org.apache.hadoop.hdfs.DFSOutputStream.hflush(DFSOutputStream.java:1946) at org.apache.hadoop.fs.FSDataOutputStream.hflush(FSDataOutputStream.java:130) ... 17 more 16/07/13 10:55:08 INFO BwaInterpreter: JMAbuin:: No sort with partitioning 16/07/13 10:55:08 ERROR LiveListenerBus: Listener EventLoggingListener threw an exception

jmabuin commented 8 years ago

Can you please check your logs with the following command?: yarn logs -applicationId your_application_id

Check there if you have this error: https://github.com/citiususc/SparkBWA#librarypatherror