17/08/30 11:14:12 INFO ContainerManagementProtocolProxy: Opening proxy : slave3.hdp:45454 17/08/30 11:14:12 INFO ContainerManagementProtocolProxy: Opening proxy : slave.hdp:45454 17/08/30 11:14:14 INFO YarnClusterSchedulerBackend: Registered executor NettyRpcEndpointRef(null) (slave3.hdp:49302) with ID 1 17/08/30 11:14:14 INFO BlockManagerMasterEndpoint: Registering block manager slave3.hdp:36548 with 2.7 GB RAM, BlockManagerId(1, slave3.hdp, 36548) 17/08/30 11:14:25 INFO YarnClusterSchedulerBackend: Registered executor NettyRpcEndpointRef(null) (slave.hdp:50760) with ID 2 17/08/30 11:14:25 INFO BlockManagerMasterEndpoint: Registering block manager slave.hdp:38078 with 2.7 GB RAM, BlockManagerId(2, slave.hdp, 38078) 17/08/30 11:14:25 INFO YarnClusterSchedulerBackend: SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.8 17/08/30 11:14:25 INFO YarnClusterScheduler: YarnClusterScheduler.postStartHook done 17/08/30 11:14:25 ERROR ApplicationMaster: User class threw exception: java.lang.OutOfMemoryError: Java heap space java.lang.OutOfMemoryError: Java heap space at cs.ucla.edu.bwaspark.datatype.BinaryFileReadUtil$.readIntArray(BinaryFileReadUtil.scala:127) at cs.ucla.edu.bwaspark.datatype.BWTType.BWTLoad(BWTType.scala:165) at cs.ucla.edu.bwaspark.datatype.BWTType.load(BWTType.scala:54) at cs.ucla.edu.bwaspark.datatype.BWAIdxType.load(BWAIdxType.scala:58) at cs.ucla.edu.bwaspark.FastMap$.memMain(FastMap.scala:119) at cs.ucla.edu.bwaspark.BWAMEMSpark$.main(BWAMEMSpark.scala:302) at cs.ucla.edu.bwaspark.BWAMEMSpark.main(BWAMEMSpark.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:559) 17/08/30 11:14:25 INFO ApplicationMaster: Final app status: FAILED, exitCode: 15, (reason: User class threw exception: java.lang.OutOfMemoryError: Java heap space) 17/08/30 11:14:25 INFO SparkContext: Invoking stop() from shutdown hook 17/08/30 11:14:25 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/metrics/json,null} 17/08/30 11:14:25 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/stage/kill,null} 17/08/30 11:14:25 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/api,null} 17/08/30 11:14:25 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/,null} 17/08/30 11:14:25 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/static,null} 17/08/30 11:14:25 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/executors/threadDump/json,null} 17/08/30 11:14:25 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/executors/threadDump,null} 17/08/30 11:14:25 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/executors/json,null} 17/08/30 11:14:25 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/executors,null} 17/08/30 11:14:25 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/environment/json,null} 17/08/30 11:14:25 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/environment,null} 17/08/30 11:14:25 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/storage/rdd/json,null} 17/08/30 11:14:25 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/storage/rdd,null} 17/08/30 11:14:25 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/storage/json,null} 17/08/30 11:14:25 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/storage,null} 17/08/30 11:14:25 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/pool/json,null} 17/08/30 11:14:25 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/pool,null} 17/08/30 11:14:25 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/stage/json,null} 17/08/30 11:14:25 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/stage,null} 17/08/30 11:14:25 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/json,null} 17/08/30 11:14:25 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages,null} 17/08/30 11:14:25 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/jobs/job/json,null} 17/08/30 11:14:25 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/jobs/job,null} 17/08/30 11:14:25 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/jobs/json,null} 17/08/30 11:14:25 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/jobs,null} 17/08/30 11:14:25 INFO SparkUI: Stopped Spark web UI at http://172.16.2.124:46846 17/08/30 11:14:25 INFO YarnAllocator: Driver requested a total number of 0 executor(s). 17/08/30 11:14:25 INFO YarnClusterSchedulerBackend: Shutting down all executors 17/08/30 11:14:25 INFO YarnClusterSchedulerBackend: Asking each executor to shut down 17/08/30 11:14:25 INFO SchedulerExtensionServices: Stopping SchedulerExtensionServices (serviceOption=None, services=List(), started=false) 17/08/30 11:14:25 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped! 17/08/30 11:14:25 INFO MemoryStore: MemoryStore cleared 17/08/30 11:14:25 INFO BlockManager: BlockManager stopped 17/08/30 11:14:25 INFO BlockManagerMaster: BlockManagerMaster stopped 17/08/30 11:14:25 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped! 17/08/30 11:14:25 INFO SparkContext: Successfully stopped SparkContext 17/08/30 11:14:25 INFO RemoteActorRefProvider$RemotingTerminator: Shutting down remote daemon. 17/08/30 11:14:25 INFO ShutdownHookManager: Shutdown hook called 17/08/30 11:14:25 INFO ShutdownHookManager: Deleting directory /var/hadoop/yarn/local/usercache/root/appcache/application_1504003590244_0009/spark-4d5f6ef5-eaf8-4de2-9371-6018e1569a9b 17/08/30 11:14:25 INFO RemoteActorRefProvider$RemotingTerminator: Remote daemon shut down; proceeding with flushing remote transports.
I'm getting this error while submitting the job
The command is ,
spark-submit --master yarn --deploy-mode cluster --driver-memory 2g --executor-memory 4g --class cs.ucla.edu.bwaspark.BWAMEMSpark --driver-java-options -Xmx4G /opt/cloud-scale-bwamem/target/cloud-scale-bwamem-0.2.2-assembly.jar cs-bwamem -bfn 4 -bPSW 1 -sbatch 10 -bPSWJNI 1 -oChoice 2 -oPath hdfs://master.hdp:8020/Files/14.adam -localRef 1 -R "@RGID:HCC1954LB:HCC1954SM:HCC1954" -isSWExtBatched 1 1 /Data/HumanBase/hg19/hg19.fa hdfs://master.hdp:8020/genomics/SampleFastqFile1.fastqThe error which i got,
17/08/30 11:14:12 INFO ContainerManagementProtocolProxy: Opening proxy : slave3.hdp:45454 17/08/30 11:14:12 INFO ContainerManagementProtocolProxy: Opening proxy : slave.hdp:45454 17/08/30 11:14:14 INFO YarnClusterSchedulerBackend: Registered executor NettyRpcEndpointRef(null) (slave3.hdp:49302) with ID 1 17/08/30 11:14:14 INFO BlockManagerMasterEndpoint: Registering block manager slave3.hdp:36548 with 2.7 GB RAM, BlockManagerId(1, slave3.hdp, 36548) 17/08/30 11:14:25 INFO YarnClusterSchedulerBackend: Registered executor NettyRpcEndpointRef(null) (slave.hdp:50760) with ID 2 17/08/30 11:14:25 INFO BlockManagerMasterEndpoint: Registering block manager slave.hdp:38078 with 2.7 GB RAM, BlockManagerId(2, slave.hdp, 38078) 17/08/30 11:14:25 INFO YarnClusterSchedulerBackend: SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.8 17/08/30 11:14:25 INFO YarnClusterScheduler: YarnClusterScheduler.postStartHook done 17/08/30 11:14:25 ERROR ApplicationMaster: User class threw exception: java.lang.OutOfMemoryError: Java heap space java.lang.OutOfMemoryError: Java heap space at cs.ucla.edu.bwaspark.datatype.BinaryFileReadUtil$.readIntArray(BinaryFileReadUtil.scala:127) at cs.ucla.edu.bwaspark.datatype.BWTType.BWTLoad(BWTType.scala:165) at cs.ucla.edu.bwaspark.datatype.BWTType.load(BWTType.scala:54) at cs.ucla.edu.bwaspark.datatype.BWAIdxType.load(BWAIdxType.scala:58) at cs.ucla.edu.bwaspark.FastMap$.memMain(FastMap.scala:119) at cs.ucla.edu.bwaspark.BWAMEMSpark$.main(BWAMEMSpark.scala:302) at cs.ucla.edu.bwaspark.BWAMEMSpark.main(BWAMEMSpark.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:559) 17/08/30 11:14:25 INFO ApplicationMaster: Final app status: FAILED, exitCode: 15, (reason: User class threw exception: java.lang.OutOfMemoryError: Java heap space) 17/08/30 11:14:25 INFO SparkContext: Invoking stop() from shutdown hook 17/08/30 11:14:25 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/metrics/json,null} 17/08/30 11:14:25 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/stage/kill,null} 17/08/30 11:14:25 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/api,null} 17/08/30 11:14:25 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/,null} 17/08/30 11:14:25 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/static,null} 17/08/30 11:14:25 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/executors/threadDump/json,null} 17/08/30 11:14:25 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/executors/threadDump,null} 17/08/30 11:14:25 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/executors/json,null} 17/08/30 11:14:25 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/executors,null} 17/08/30 11:14:25 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/environment/json,null} 17/08/30 11:14:25 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/environment,null} 17/08/30 11:14:25 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/storage/rdd/json,null} 17/08/30 11:14:25 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/storage/rdd,null} 17/08/30 11:14:25 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/storage/json,null} 17/08/30 11:14:25 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/storage,null} 17/08/30 11:14:25 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/pool/json,null} 17/08/30 11:14:25 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/pool,null} 17/08/30 11:14:25 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/stage/json,null} 17/08/30 11:14:25 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/stage,null} 17/08/30 11:14:25 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/json,null} 17/08/30 11:14:25 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages,null} 17/08/30 11:14:25 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/jobs/job/json,null} 17/08/30 11:14:25 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/jobs/job,null} 17/08/30 11:14:25 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/jobs/json,null} 17/08/30 11:14:25 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/jobs,null} 17/08/30 11:14:25 INFO SparkUI: Stopped Spark web UI at http://172.16.2.124:46846 17/08/30 11:14:25 INFO YarnAllocator: Driver requested a total number of 0 executor(s). 17/08/30 11:14:25 INFO YarnClusterSchedulerBackend: Shutting down all executors 17/08/30 11:14:25 INFO YarnClusterSchedulerBackend: Asking each executor to shut down 17/08/30 11:14:25 INFO SchedulerExtensionServices: Stopping SchedulerExtensionServices (serviceOption=None, services=List(), started=false) 17/08/30 11:14:25 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped! 17/08/30 11:14:25 INFO MemoryStore: MemoryStore cleared 17/08/30 11:14:25 INFO BlockManager: BlockManager stopped 17/08/30 11:14:25 INFO BlockManagerMaster: BlockManagerMaster stopped 17/08/30 11:14:25 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped! 17/08/30 11:14:25 INFO SparkContext: Successfully stopped SparkContext 17/08/30 11:14:25 INFO RemoteActorRefProvider$RemotingTerminator: Shutting down remote daemon. 17/08/30 11:14:25 INFO ShutdownHookManager: Shutdown hook called 17/08/30 11:14:25 INFO ShutdownHookManager: Deleting directory /var/hadoop/yarn/local/usercache/root/appcache/application_1504003590244_0009/spark-4d5f6ef5-eaf8-4de2-9371-6018e1569a9b 17/08/30 11:14:25 INFO RemoteActorRefProvider$RemotingTerminator: Remote daemon shut down; proceeding with flushing remote transports.