Worker is listed in the removed executors list with its state as KILLED.
However this is an output file being produced, it is not clear yet if this is a real problem.
Steps to reproduce
adamcloud@Adams-Mac-mini-Ubuntu:~$ docker exec adam adam-submit --master spark://spark-master:7077 transform hdfs://hdfs-namenode:9000/SRR062634.sam hdfs://hdfs-namenode:9000/SRR062634b.adam
Spark assembly has been built with Hive, including Datanucleus jars on classpath
2015-04-14 18:56:10 WARN NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
STDERR
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
15/04/14 18:56:11 INFO CoarseGrainedExecutorBackend: Registered signal handlers for [TERM, HUP, INT]
15/04/14 18:56:12 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
15/04/14 18:56:12 INFO SecurityManager: Changing view acls to: root
15/04/14 18:56:12 INFO SecurityManager: Changing modify acls to: root
15/04/14 18:56:12 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root); users with modify permissions: Set(root)
15/04/14 18:56:12 INFO Slf4jLogger: Slf4jLogger started
15/04/14 18:56:12 INFO Remoting: Starting remoting
15/04/14 18:56:12 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://driverPropsFetcher@spark-worker1:60308]
15/04/14 18:56:12 INFO Remoting: Remoting now listens on addresses: [akka.tcp://driverPropsFetcher@spark-worker1:60308]
15/04/14 18:56:12 INFO Utils: Successfully started service 'driverPropsFetcher' on port 60308.
15/04/14 18:56:12 INFO SecurityManager: Changing view acls to: root
15/04/14 18:56:12 INFO SecurityManager: Changing modify acls to: root
15/04/14 18:56:12 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root); users with modify permissions: Set(root)
15/04/14 18:56:12 INFO RemoteActorRefProvider$RemotingTerminator: Shutting down remote daemon.
15/04/14 18:56:12 INFO RemoteActorRefProvider$RemotingTerminator: Remote daemon shut down; proceeding with flushing remote transports.
15/04/14 18:56:12 INFO Slf4jLogger: Slf4jLogger started
15/04/14 18:56:12 INFO Remoting: Starting remoting
15/04/14 18:56:12 INFO Remoting: Remoting shut down
15/04/14 18:56:12 INFO RemoteActorRefProvider$RemotingTerminator: Remoting shut down.
15/04/14 18:56:12 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkExecutor@spark-worker1:37876]
15/04/14 18:56:12 INFO Remoting: Remoting now listens on addresses: [akka.tcp://sparkExecutor@spark-worker1:37876]
15/04/14 18:56:12 INFO Utils: Successfully started service 'sparkExecutor' on port 37876.
15/04/14 18:56:12 INFO CoarseGrainedExecutorBackend: Connecting to driver: akka.tcp://sparkDriver@adam:33997/user/CoarseGrainedScheduler
15/04/14 18:56:12 INFO WorkerWatcher: Connecting to worker akka.tcp://sparkWorker@spark-worker1:43766/user/Worker
15/04/14 18:56:12 INFO WorkerWatcher: Successfully connected to akka.tcp://sparkWorker@spark-worker1:43766/user/Worker
15/04/14 18:56:12 INFO CoarseGrainedExecutorBackend: Successfully registered with driver
15/04/14 18:56:12 INFO SecurityManager: Changing view acls to: root
15/04/14 18:56:12 INFO SecurityManager: Changing modify acls to: root
15/04/14 18:56:12 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root); users with modify permissions: Set(root)
15/04/14 18:56:12 INFO Slf4jLogger: Slf4jLogger started
15/04/14 18:56:12 INFO Remoting: Starting remoting
15/04/14 18:56:13 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkExecutor@spark-worker1:46753]
15/04/14 18:56:13 INFO Remoting: Remoting now listens on addresses: [akka.tcp://sparkExecutor@spark-worker1:46753]
15/04/14 18:56:13 INFO Utils: Successfully started service 'sparkExecutor' on port 46753.
15/04/14 18:56:13 INFO AkkaUtils: Connecting to MapOutputTracker: akka.tcp://sparkDriver@adam:33997/user/MapOutputTracker
15/04/14 18:56:13 INFO AkkaUtils: Connecting to BlockManagerMaster: akka.tcp://sparkDriver@adam:33997/user/BlockManagerMaster
15/04/14 18:56:13 INFO DiskBlockManager: Created local directory at /tmp/spark-local-20150414185613-f576
15/04/14 18:56:13 INFO Utils: Successfully started service 'Connection manager for block manager' on port 37422.
15/04/14 18:56:13 INFO ConnectionManager: Bound socket to port 37422 with id = ConnectionManagerId(spark-worker1,37422)
15/04/14 18:56:13 INFO MemoryStore: MemoryStore started with capacity 1060.3 MB
15/04/14 18:56:13 INFO BlockManagerMaster: Trying to register BlockManager
15/04/14 18:56:13 INFO BlockManagerMaster: Registered BlockManager
15/04/14 18:56:13 INFO AkkaUtils: Connecting to HeartbeatReceiver: akka.tcp://sparkDriver@adam:33997/user/HeartbeatReceiver
15/04/14 18:56:13 INFO CoarseGrainedExecutorBackend: Got assigned task 0
15/04/14 18:56:13 INFO Executor: Running task 0.0 in stage 0.0 (TID 0)
15/04/14 18:56:13 INFO Executor: Fetching http://172.17.0.16:45360/jars/paranamer-2.3.jar with timestamp 1429037770711
15/04/14 18:56:13 INFO Utils: Fetching http://172.17.0.16:45360/jars/paranamer-2.3.jar to /tmp/fetchFileTemp9138374815620003401.tmp
15/04/14 18:56:13 INFO Executor: Adding file:/usr/local/spark-1.1.0-bin-hadoop2.3/work/app-20150414185611-0000/0/./paranamer-2.3.jar to class loader
15/04/14 18:56:13 INFO Executor: Fetching http://172.17.0.16:45360/jars/commons-cli-1.2.jar with timestamp 1429037770686
15/04/14 18:56:13 INFO Utils: Fetching http://172.17.0.16:45360/jars/commons-cli-1.2.jar to /tmp/fetchFileTemp4300381861535729741.tmp
15/04/14 18:56:13 INFO Executor: Adding file:/usr/local/spark-1.1.0-bin-hadoop2.3/work/app-20150414185611-0000/0/./commons-cli-1.2.jar to class loader
15/04/14 18:56:13 INFO Executor: Fetching http://172.17.0.16:45360/jars/commons-httpclient-3.1.jar with timestamp 1429037770688
15/04/14 18:56:13 INFO Utils: Fetching http://172.17.0.16:45360/jars/commons-httpclient-3.1.jar to /tmp/fetchFileTemp2955185790565174009.tmp
15/04/14 18:56:13 INFO Executor: Adding file:/usr/local/spark-1.1.0-bin-hadoop2.3/work/app-20150414185611-0000/0/./commons-httpclient-3.1.jar to class loader
15/04/14 18:56:13 INFO Executor: Fetching http://172.17.0.16:45360/jars/jcommander-1.27.jar with timestamp 1429037770786
15/04/14 18:56:13 INFO Utils: Fetching http://172.17.0.16:45360/jars/jcommander-1.27.jar to /tmp/fetchFileTemp3393129393988809981.tmp
15/04/14 18:56:13 INFO Executor: Adding file:/usr/local/spark-1.1.0-bin-hadoop2.3/work/app-20150414185611-0000/0/./jcommander-1.27.jar to class loader
15/04/14 18:56:13 INFO Executor: Fetching http://172.17.0.16:45360/jars/objenesis-1.2.jar with timestamp 1429037770720
15/04/14 18:56:13 INFO Utils: Fetching http://172.17.0.16:45360/jars/objenesis-1.2.jar to /tmp/fetchFileTemp5267878146906553193.tmp
15/04/14 18:56:13 INFO Executor: Adding file:/usr/local/spark-1.1.0-bin-hadoop2.3/work/app-20150414185611-0000/0/./objenesis-1.2.jar to class loader
15/04/14 18:56:13 INFO Executor: Fetching http://172.17.0.16:45360/jars/fastutil-6.4.4.jar with timestamp 1429037770755
15/04/14 18:56:13 INFO Utils: Fetching http://172.17.0.16:45360/jars/fastutil-6.4.4.jar to /tmp/fetchFileTemp1363741256222612084.tmp
15/04/14 18:56:13 INFO Executor: Adding file:/usr/local/spark-1.1.0-bin-hadoop2.3/work/app-20150414185611-0000/0/./fastutil-6.4.4.jar to class loader
15/04/14 18:56:13 INFO Executor: Fetching http://172.17.0.16:45360/jars/commons-jexl-2.1.1.jar with timestamp 1429037770782
15/04/14 18:56:13 INFO Utils: Fetching http://172.17.0.16:45360/jars/commons-jexl-2.1.1.jar to /tmp/fetchFileTemp4854967959179966282.tmp
15/04/14 18:56:13 INFO Executor: Adding file:/usr/local/spark-1.1.0-bin-hadoop2.3/work/app-20150414185611-0000/0/./commons-jexl-2.1.1.jar to class loader
15/04/14 18:56:13 INFO Executor: Fetching http://172.17.0.16:45360/jars/grizzled-slf4j_2.10-1.0.2.jar with timestamp 1429037770856
15/04/14 18:56:13 INFO Utils: Fetching http://172.17.0.16:45360/jars/grizzled-slf4j_2.10-1.0.2.jar to /tmp/fetchFileTemp2750787710904710876.tmp
15/04/14 18:56:13 INFO Executor: Adding file:/usr/local/spark-1.1.0-bin-hadoop2.3/work/app-20150414185611-0000/0/./grizzled-slf4j_2.10-1.0.2.jar to class loader
15/04/14 18:56:13 INFO Executor: Fetching http://172.17.0.16:45360/jars/parquet-jackson-1.6.0rc4.jar with timestamp 1429037770764
15/04/14 18:56:13 INFO Utils: Fetching http://172.17.0.16:45360/jars/parquet-jackson-1.6.0rc4.jar to /tmp/fetchFileTemp7793053302073882282.tmp
15/04/14 18:56:13 INFO Executor: Adding file:/usr/local/spark-1.1.0-bin-hadoop2.3/work/app-20150414185611-0000/0/./parquet-jackson-1.6.0rc4.jar to class loader
15/04/14 18:56:13 INFO Executor: Fetching http://172.17.0.16:45360/jars/bcel-5.2.jar with timestamp 1429037770781
15/04/14 18:56:13 INFO Utils: Fetching http://172.17.0.16:45360/jars/bcel-5.2.jar to /tmp/fetchFileTemp6684520750287603408.tmp
15/04/14 18:56:13 INFO Executor: Adding file:/usr/local/spark-1.1.0-bin-hadoop2.3/work/app-20150414185611-0000/0/./bcel-5.2.jar to class loader
15/04/14 18:56:13 INFO Executor: Fetching http://172.17.0.16:45360/jars/hadoop-bam-7.0.0.jar with timestamp 1429037770767
15/04/14 18:56:13 INFO Utils: Fetching http://172.17.0.16:45360/jars/hadoop-bam-7.0.0.jar to /tmp/fetchFileTemp4154138373923122267.tmp
15/04/14 18:56:13 INFO Executor: Adding file:/usr/local/spark-1.1.0-bin-hadoop2.3/work/app-20150414185611-0000/0/./hadoop-bam-7.0.0.jar to class loader
15/04/14 18:56:13 INFO Executor: Fetching http://172.17.0.16:45360/jars/servo-core-0.5.5.jar with timestamp 1429037770791
15/04/14 18:56:13 INFO Utils: Fetching http://172.17.0.16:45360/jars/servo-core-0.5.5.jar to /tmp/fetchFileTemp9045351472743461951.tmp
15/04/14 18:56:13 INFO Executor: Adding file:/usr/local/spark-1.1.0-bin-hadoop2.3/work/app-20150414185611-0000/0/./servo-core-0.5.5.jar to class loader
15/04/14 18:56:13 INFO Executor: Fetching http://172.17.0.16:45360/jars/snappy-java-1.0.5.3.jar with timestamp 1429037770703
15/04/14 18:56:13 INFO Utils: Fetching http://172.17.0.16:45360/jars/snappy-java-1.0.5.3.jar to /tmp/fetchFileTemp6129776536072113626.tmp
15/04/14 18:56:13 INFO Executor: Adding file:/usr/local/spark-1.1.0-bin-hadoop2.3/work/app-20150414185611-0000/0/./snappy-java-1.0.5.3.jar to class loader
15/04/14 18:56:13 INFO Executor: Fetching http://172.17.0.16:45360/jars/cofoja-1.1-r150.jar with timestamp 1429037770770
15/04/14 18:56:13 INFO Utils: Fetching http://172.17.0.16:45360/jars/cofoja-1.1-r150.jar to /tmp/fetchFileTemp7506307979723596237.tmp
15/04/14 18:56:13 INFO Executor: Adding file:/usr/local/spark-1.1.0-bin-hadoop2.3/work/app-20150414185611-0000/0/./cofoja-1.1-r150.jar to class loader
15/04/14 18:56:13 INFO Executor: Fetching http://172.17.0.16:45360/jars/json4s-ast_2.10-3.2.10.jar with timestamp 1429037770849
15/04/14 18:56:13 INFO Utils: Fetching http://172.17.0.16:45360/jars/json4s-ast_2.10-3.2.10.jar to /tmp/fetchFileTemp5860121585304136161.tmp
15/04/14 18:56:13 INFO Executor: Adding file:/usr/local/spark-1.1.0-bin-hadoop2.3/work/app-20150414185611-0000/0/./json4s-ast_2.10-3.2.10.jar to class loader
15/04/14 18:56:13 INFO Executor: Fetching http://172.17.0.16:45360/jars/scala-library-2.10.4.jar with timestamp 1429037770808
15/04/14 18:56:13 INFO Utils: Fetching http://172.17.0.16:45360/jars/scala-library-2.10.4.jar to /tmp/fetchFileTemp4255564728918789950.tmp
15/04/14 18:56:13 INFO Executor: Adding file:/usr/local/spark-1.1.0-bin-hadoop2.3/work/app-20150414185611-0000/0/./scala-library-2.10.4.jar to class loader
15/04/14 18:56:13 INFO Executor: Fetching http://172.17.0.16:45360/jars/parquet-format-2.2.0-rc1.jar with timestamp 1429037770766
15/04/14 18:56:13 INFO Utils: Fetching http://172.17.0.16:45360/jars/parquet-format-2.2.0-rc1.jar to /tmp/fetchFileTemp165001245190906794.tmp
15/04/14 18:56:13 INFO Executor: Adding file:/usr/local/spark-1.1.0-bin-hadoop2.3/work/app-20150414185611-0000/0/./parquet-format-2.2.0-rc1.jar to class loader
15/04/14 18:56:13 INFO Executor: Fetching http://172.17.0.16:45360/jars/commons-io-1.3.2.jar with timestamp 1429037770704
15/04/14 18:56:13 INFO Utils: Fetching http://172.17.0.16:45360/jars/commons-io-1.3.2.jar to /tmp/fetchFileTemp5067147027478568727.tmp
15/04/14 18:56:13 INFO Executor: Adding file:/usr/local/spark-1.1.0-bin-hadoop2.3/work/app-20150414185611-0000/0/./commons-io-1.3.2.jar to class loader
15/04/14 18:56:13 INFO Executor: Fetching http://172.17.0.16:45360/jars/asm-4.0.jar with timestamp 1429037770720
15/04/14 18:56:13 INFO Utils: Fetching http://172.17.0.16:45360/jars/asm-4.0.jar to /tmp/fetchFileTemp318227717983144312.tmp
15/04/14 18:56:13 INFO Executor: Adding file:/usr/local/spark-1.1.0-bin-hadoop2.3/work/app-20150414185611-0000/0/./asm-4.0.jar to class loader
15/04/14 18:56:13 INFO Executor: Fetching http://172.17.0.16:45360/jars/commons-compress-1.4.1.jar with timestamp 1429037770690
15/04/14 18:56:13 INFO Utils: Fetching http://172.17.0.16:45360/jars/commons-compress-1.4.1.jar to /tmp/fetchFileTemp4348044553948052935.tmp
15/04/14 18:56:13 INFO Executor: Adding file:/usr/local/spark-1.1.0-bin-hadoop2.3/work/app-20150414185611-0000/0/./commons-compress-1.4.1.jar to class loader
15/04/14 18:56:13 INFO Executor: Fetching http://172.17.0.16:45360/jars/slf4j-log4j12-1.7.5.jar with timestamp 1429037770809
15/04/14 18:56:13 INFO Utils: Fetching http://172.17.0.16:45360/jars/slf4j-log4j12-1.7.5.jar to /tmp/fetchFileTemp1260749043253037096.tmp
15/04/14 18:56:13 INFO Executor: Adding file:/usr/local/spark-1.1.0-bin-hadoop2.3/work/app-20150414185611-0000/0/./slf4j-log4j12-1.7.5.jar to class loader
15/04/14 18:56:13 INFO Executor: Fetching http://172.17.0.16:45360/jars/kryo-2.21.jar with timestamp 1429037770719
15/04/14 18:56:13 INFO Utils: Fetching http://172.17.0.16:45360/jars/kryo-2.21.jar to /tmp/fetchFileTemp7213170923379151436.tmp
15/04/14 18:56:13 INFO Executor: Adding file:/usr/local/spark-1.1.0-bin-hadoop2.3/work/app-20150414185611-0000/0/./kryo-2.21.jar to class loader
15/04/14 18:56:13 INFO Executor: Fetching http://172.17.0.16:45360/jars/args4j-2.0.23.jar with timestamp 1429037770809
15/04/14 18:56:13 INFO Utils: Fetching http://172.17.0.16:45360/jars/args4j-2.0.23.jar to /tmp/fetchFileTemp8135928262602238416.tmp
15/04/14 18:56:13 INFO Executor: Adding file:/usr/local/spark-1.1.0-bin-hadoop2.3/work/app-20150414185611-0000/0/./args4j-2.0.23.jar to class loader
15/04/14 18:56:13 INFO Executor: Fetching http://172.17.0.16:45360/jars/parquet-hadoop-1.6.0rc4.jar with timestamp 1429037770761
15/04/14 18:56:13 INFO Utils: Fetching http://172.17.0.16:45360/jars/parquet-hadoop-1.6.0rc4.jar to /tmp/fetchFileTemp4657446138937067937.tmp
15/04/14 18:56:13 INFO Executor: Adding file:/usr/local/spark-1.1.0-bin-hadoop2.3/work/app-20150414185611-0000/0/./parquet-hadoop-1.6.0rc4.jar to class loader
15/04/14 18:56:13 INFO Executor: Fetching http://172.17.0.16:45360/jars/joda-convert-1.6.jar with timestamp 1429037770861
15/04/14 18:56:13 INFO Utils: Fetching http://172.17.0.16:45360/jars/joda-convert-1.6.jar to /tmp/fetchFileTemp5262321399304079929.tmp
15/04/14 18:56:13 INFO Executor: Adding file:/usr/local/spark-1.1.0-bin-hadoop2.3/work/app-20150414185611-0000/0/./joda-convert-1.6.jar to class loader
15/04/14 18:56:13 INFO Executor: Fetching http://172.17.0.16:45360/jars/parquet-avro-1.6.0rc4.jar with timestamp 1429037770756
15/04/14 18:56:13 INFO Utils: Fetching http://172.17.0.16:45360/jars/parquet-avro-1.6.0rc4.jar to /tmp/fetchFileTemp328522683892592644.tmp
15/04/14 18:56:13 INFO Executor: Adding file:/usr/local/spark-1.1.0-bin-hadoop2.3/work/app-20150414185611-0000/0/./parquet-avro-1.6.0rc4.jar to class loader
15/04/14 18:56:13 INFO Executor: Fetching http://172.17.0.16:45360/jars/scalatra-json_2.10-2.3.0.jar with timestamp 1429037770846
15/04/14 18:56:13 INFO Utils: Fetching http://172.17.0.16:45360/jars/scalatra-json_2.10-2.3.0.jar to /tmp/fetchFileTemp891212980175908144.tmp
15/04/14 18:56:13 INFO Executor: Adding file:/usr/local/spark-1.1.0-bin-hadoop2.3/work/app-20150414185611-0000/0/./scalatra-json_2.10-2.3.0.jar to class loader
15/04/14 18:56:13 INFO Executor: Fetching http://172.17.0.16:45360/jars/scala-reflect-2.10.4.jar with timestamp 1429037770867
15/04/14 18:56:13 INFO Utils: Fetching http://172.17.0.16:45360/jars/scala-reflect-2.10.4.jar to /tmp/fetchFileTemp27874364891379654.tmp
15/04/14 18:56:13 INFO Executor: Adding file:/usr/local/spark-1.1.0-bin-hadoop2.3/work/app-20150414185611-0000/0/./scala-reflect-2.10.4.jar to class loader
15/04/14 18:56:13 INFO Executor: Fetching http://172.17.0.16:45360/jars/jackson-mapper-asl-1.9.13.jar with timestamp 1429037770710
15/04/14 18:56:13 INFO Utils: Fetching http://172.17.0.16:45360/jars/jackson-mapper-asl-1.9.13.jar to /tmp/fetchFileTemp8180814965879888474.tmp
15/04/14 18:56:13 INFO Executor: Adding file:/usr/local/spark-1.1.0-bin-hadoop2.3/work/app-20150414185611-0000/0/./jackson-mapper-asl-1.9.13.jar to class loader
15/04/14 18:56:13 INFO Executor: Fetching http://172.17.0.16:45360/jars/scala-compiler-2.10.0.jar with timestamp 1429037770846
15/04/14 18:56:13 INFO Utils: Fetching http://172.17.0.16:45360/jars/scala-compiler-2.10.0.jar to /tmp/fetchFileTemp6701382586835769484.tmp
15/04/14 18:56:13 INFO Executor: Adding file:/usr/local/spark-1.1.0-bin-hadoop2.3/work/app-20150414185611-0000/0/./scala-compiler-2.10.0.jar to class loader
15/04/14 18:56:13 INFO Executor: Fetching http://172.17.0.16:45360/jars/bsh-2.0b4.jar with timestamp 1429037770786
15/04/14 18:56:13 INFO Utils: Fetching http://172.17.0.16:45360/jars/bsh-2.0b4.jar to /tmp/fetchFileTemp2388925456893403133.tmp
15/04/14 18:56:13 INFO Executor: Adding file:/usr/local/spark-1.1.0-bin-hadoop2.3/work/app-20150414185611-0000/0/./bsh-2.0b4.jar to class loader
15/04/14 18:56:13 INFO Executor: Fetching http://172.17.0.16:45360/jars/rl_2.10-0.4.10.jar with timestamp 1429037770857
15/04/14 18:56:13 INFO Utils: Fetching http://172.17.0.16:45360/jars/rl_2.10-0.4.10.jar to /tmp/fetchFileTemp7265779172490063278.tmp
15/04/14 18:56:13 INFO Executor: Adding file:/usr/local/spark-1.1.0-bin-hadoop2.3/work/app-20150414185611-0000/0/./rl_2.10-0.4.10.jar to class loader
15/04/14 18:56:13 INFO Executor: Fetching http://172.17.0.16:45360/jars/minlog-1.2.jar with timestamp 1429037770720
15/04/14 18:56:13 INFO Utils: Fetching http://172.17.0.16:45360/jars/minlog-1.2.jar to /tmp/fetchFileTemp7393705099077145723.tmp
15/04/14 18:56:13 INFO Executor: Adding file:/usr/local/spark-1.1.0-bin-hadoop2.3/work/app-20150414185611-0000/0/./minlog-1.2.jar to class loader
15/04/14 18:56:13 INFO Executor: Fetching http://172.17.0.16:45360/jars/mime-util-2.1.3.jar with timestamp 1429037770859
15/04/14 18:56:13 INFO Utils: Fetching http://172.17.0.16:45360/jars/mime-util-2.1.3.jar to /tmp/fetchFileTemp304383920554853460.tmp
15/04/14 18:56:13 INFO Executor: Adding file:/usr/local/spark-1.1.0-bin-hadoop2.3/work/app-20150414185611-0000/0/./mime-util-2.1.3.jar to class loader
15/04/14 18:56:13 INFO Executor: Fetching http://172.17.0.16:45360/jars/scalap-2.10.0.jar with timestamp 1429037770851
15/04/14 18:56:13 INFO Utils: Fetching http://172.17.0.16:45360/jars/scalap-2.10.0.jar to /tmp/fetchFileTemp4091359871699194773.tmp
15/04/14 18:56:13 INFO Executor: Adding file:/usr/local/spark-1.1.0-bin-hadoop2.3/work/app-20150414185611-0000/0/./scalap-2.10.0.jar to class loader
15/04/14 18:56:13 INFO Executor: Fetching http://172.17.0.16:45360/jars/xz-1.0.jar with timestamp 1429037770690
15/04/14 18:56:13 INFO Utils: Fetching http://172.17.0.16:45360/jars/xz-1.0.jar to /tmp/fetchFileTemp1898496628024199440.tmp
15/04/14 18:56:13 INFO Executor: Adding file:/usr/local/spark-1.1.0-bin-hadoop2.3/work/app-20150414185611-0000/0/./xz-1.0.jar to class loader
15/04/14 18:56:13 INFO Executor: Fetching http://172.17.0.16:45360/jars/httpcore-4.3.1.jar with timestamp 1429037770790
15/04/14 18:56:13 INFO Utils: Fetching http://172.17.0.16:45360/jars/httpcore-4.3.1.jar to /tmp/fetchFileTemp7892039994601518338.tmp
15/04/14 18:56:13 INFO Executor: Adding file:/usr/local/spark-1.1.0-bin-hadoop2.3/work/app-20150414185611-0000/0/./httpcore-4.3.1.jar to class loader
15/04/14 18:56:13 INFO Executor: Fetching http://172.17.0.16:45360/jars/jakarta-regexp-1.4.jar with timestamp 1429037770781
15/04/14 18:56:13 INFO Utils: Fetching http://172.17.0.16:45360/jars/jakarta-regexp-1.4.jar to /tmp/fetchFileTemp1426692907134838681.tmp
15/04/14 18:56:13 INFO Executor: Adding file:/usr/local/spark-1.1.0-bin-hadoop2.3/work/app-20150414185611-0000/0/./jakarta-regexp-1.4.jar to class loader
15/04/14 18:56:13 INFO Executor: Fetching http://172.17.0.16:45360/jars/parquet-common-1.6.0rc4.jar with timestamp 1429037770759
15/04/14 18:56:13 INFO Utils: Fetching http://172.17.0.16:45360/jars/parquet-common-1.6.0rc4.jar to /tmp/fetchFileTemp5086786835973030426.tmp
15/04/14 18:56:13 INFO Executor: Adding file:/usr/local/spark-1.1.0-bin-hadoop2.3/work/app-20150414185611-0000/0/./parquet-common-1.6.0rc4.jar to class loader
15/04/14 18:56:13 INFO Executor: Fetching http://172.17.0.16:45360/jars/slf4j-api-1.7.5.jar with timestamp 1429037770697
15/04/14 18:56:13 INFO Utils: Fetching http://172.17.0.16:45360/jars/slf4j-api-1.7.5.jar to /tmp/fetchFileTemp6896817311355197448.tmp
15/04/14 18:56:13 INFO Executor: Adding file:/usr/local/spark-1.1.0-bin-hadoop2.3/work/app-20150414185611-0000/0/./slf4j-api-1.7.5.jar to class loader
15/04/14 18:56:13 INFO Executor: Fetching http://172.17.0.16:45360/jars/log4j-1.2.17.jar with timestamp 1429037770698
15/04/14 18:56:13 INFO Utils: Fetching http://172.17.0.16:45360/jars/log4j-1.2.17.jar to /tmp/fetchFileTemp4527060373665486864.tmp
15/04/14 18:56:13 INFO Executor: Adding file:/usr/local/spark-1.1.0-bin-hadoop2.3/work/app-20150414185611-0000/0/./log4j-1.2.17.jar to class loader
15/04/14 18:56:13 INFO Executor: Fetching http://172.17.0.16:45360/jars/ant-1.8.2.jar with timestamp 1429037770778
15/04/14 18:56:13 INFO Utils: Fetching http://172.17.0.16:45360/jars/ant-1.8.2.jar to /tmp/fetchFileTemp6733947518044913252.tmp
15/04/14 18:56:13 INFO Executor: Adding file:/usr/local/spark-1.1.0-bin-hadoop2.3/work/app-20150414185611-0000/0/./ant-1.8.2.jar to class loader
15/04/14 18:56:13 INFO Executor: Fetching http://172.17.0.16:45360/jars/parquet-generator-1.6.0rc4.jar with timestamp 1429037770760
15/04/14 18:56:13 INFO Utils: Fetching http://172.17.0.16:45360/jars/parquet-generator-1.6.0rc4.jar to /tmp/fetchFileTemp2787981656240273372.tmp
15/04/14 18:56:13 INFO Executor: Adding file:/usr/local/spark-1.1.0-bin-hadoop2.3/work/app-20150414185611-0000/0/./parquet-generator-1.6.0rc4.jar to class loader
15/04/14 18:56:13 INFO Executor: Fetching http://172.17.0.16:45360/jars/httpclient-4.3.2.jar with timestamp 1429037770788
15/04/14 18:56:13 INFO Utils: Fetching http://172.17.0.16:45360/jars/httpclient-4.3.2.jar to /tmp/fetchFileTemp767377273667406719.tmp
15/04/14 18:56:13 INFO Executor: Adding file:/usr/local/spark-1.1.0-bin-hadoop2.3/work/app-20150414185611-0000/0/./httpclient-4.3.2.jar to class loader
15/04/14 18:56:13 INFO Executor: Fetching http://172.17.0.16:45360/jars/joda-time-2.3.jar with timestamp 1429037770860
15/04/14 18:56:13 INFO Utils: Fetching http://172.17.0.16:45360/jars/joda-time-2.3.jar to /tmp/fetchFileTemp8092498449699013036.tmp
15/04/14 18:56:13 INFO Executor: Adding file:/usr/local/spark-1.1.0-bin-hadoop2.3/work/app-20150414185611-0000/0/./joda-time-2.3.jar to class loader
15/04/14 18:56:13 INFO Executor: Fetching http://172.17.0.16:45360/jars/adam-core-0.15.0.jar with timestamp 1429037770717
15/04/14 18:56:13 INFO Utils: Fetching http://172.17.0.16:45360/jars/adam-core-0.15.0.jar to /tmp/fetchFileTemp6699838462856191513.tmp
15/04/14 18:56:13 INFO Executor: Adding file:/usr/local/spark-1.1.0-bin-hadoop2.3/work/app-20150414185611-0000/0/./adam-core-0.15.0.jar to class loader
15/04/14 18:56:13 INFO Executor: Fetching http://172.17.0.16:45360/jars/htsjdk-1.118.jar with timestamp 1429037770773
15/04/14 18:56:13 INFO Utils: Fetching http://172.17.0.16:45360/jars/htsjdk-1.118.jar to /tmp/fetchFileTemp8057782358742865819.tmp
15/04/14 18:56:13 INFO Executor: Adding file:/usr/local/spark-1.1.0-bin-hadoop2.3/work/app-20150414185611-0000/0/./htsjdk-1.118.jar to class loader
15/04/14 18:56:13 INFO Executor: Fetching http://172.17.0.16:45360/jars/commons-codec-1.4.jar with timestamp 1429037770688
15/04/14 18:56:13 INFO Utils: Fetching http://172.17.0.16:45360/jars/commons-codec-1.4.jar to /tmp/fetchFileTemp2904536354338960242.tmp
15/04/14 18:56:13 INFO Executor: Adding file:/usr/local/spark-1.1.0-bin-hadoop2.3/work/app-20150414185611-0000/0/./commons-codec-1.4.jar to class loader
15/04/14 18:56:13 INFO Executor: Fetching http://172.17.0.16:45360/jars/testng-6.8.8.jar with timestamp 1429037770785
15/04/14 18:56:13 INFO Utils: Fetching http://172.17.0.16:45360/jars/testng-6.8.8.jar to /tmp/fetchFileTemp177021206433593691.tmp
15/04/14 18:56:13 INFO Executor: Adding file:/usr/local/spark-1.1.0-bin-hadoop2.3/work/app-20150414185611-0000/0/./testng-6.8.8.jar to class loader
15/04/14 18:56:13 INFO Executor: Fetching http://172.17.0.16:45360/jars/adam-cli-0.15.0.jar with timestamp 1429037770868
15/04/14 18:56:13 INFO Utils: Fetching http://172.17.0.16:45360/jars/adam-cli-0.15.0.jar to /tmp/fetchFileTemp1198597501853490421.tmp
15/04/14 18:56:13 INFO Executor: Adding file:/usr/local/spark-1.1.0-bin-hadoop2.3/work/app-20150414185611-0000/0/./adam-cli-0.15.0.jar to class loader
15/04/14 18:56:13 INFO Executor: Fetching http://172.17.0.16:45360/jars/scalate-core_2.10-1.6.1.jar with timestamp 1429037770815
15/04/14 18:56:13 INFO Utils: Fetching http://172.17.0.16:45360/jars/scalate-core_2.10-1.6.1.jar to /tmp/fetchFileTemp1544095283491947968.tmp
15/04/14 18:56:13 INFO Executor: Adding file:/usr/local/spark-1.1.0-bin-hadoop2.3/work/app-20150414185611-0000/0/./scalate-core_2.10-1.6.1.jar to class loader
15/04/14 18:56:13 INFO Executor: Fetching http://172.17.0.16:45360/jars/scalac-scoverage-plugin_2.10-0.99.2.jar with timestamp 1429037770704
15/04/14 18:56:13 INFO Utils: Fetching http://172.17.0.16:45360/jars/scalac-scoverage-plugin_2.10-0.99.2.jar to /tmp/fetchFileTemp2418724355007354737.tmp
15/04/14 18:56:13 INFO Executor: Adding file:/usr/local/spark-1.1.0-bin-hadoop2.3/work/app-20150414185611-0000/0/./scalac-scoverage-plugin_2.10-0.99.2.jar to class loader
15/04/14 18:56:13 INFO Executor: Fetching http://172.17.0.16:45360/jars/commons-logging-1.1.3.jar with timestamp 1429037770689
15/04/14 18:56:13 INFO Utils: Fetching http://172.17.0.16:45360/jars/commons-logging-1.1.3.jar to /tmp/fetchFileTemp2193111831579078333.tmp
15/04/14 18:56:13 INFO Executor: Adding file:/usr/local/spark-1.1.0-bin-hadoop2.3/work/app-20150414185611-0000/0/./commons-logging-1.1.3.jar to class loader
15/04/14 18:56:13 INFO Executor: Fetching http://172.17.0.16:45360/jars/jackson-core-asl-1.9.13.jar with timestamp 1429037770708
15/04/14 18:56:13 INFO Utils: Fetching http://172.17.0.16:45360/jars/jackson-core-asl-1.9.13.jar to /tmp/fetchFileTemp2508275494061804380.tmp
15/04/14 18:56:13 INFO Executor: Adding file:/usr/local/spark-1.1.0-bin-hadoop2.3/work/app-20150414185611-0000/0/./jackson-core-asl-1.9.13.jar to class loader
15/04/14 18:56:13 INFO Executor: Fetching http://172.17.0.16:45360/jars/avro-1.7.6.jar with timestamp 1429037770707
15/04/14 18:56:13 INFO Utils: Fetching http://172.17.0.16:45360/jars/avro-1.7.6.jar to /tmp/fetchFileTemp3201061414883736034.tmp
15/04/14 18:56:13 INFO Executor: Adding file:/usr/local/spark-1.1.0-bin-hadoop2.3/work/app-20150414185611-0000/0/./avro-1.7.6.jar to class loader
15/04/14 18:56:13 INFO Executor: Fetching http://172.17.0.16:45360/jars/ant-launcher-1.8.2.jar with timestamp 1429037770779
15/04/14 18:56:13 INFO Utils: Fetching http://172.17.0.16:45360/jars/ant-launcher-1.8.2.jar to /tmp/fetchFileTemp8671223998556853064.tmp
15/04/14 18:56:13 INFO Executor: Adding file:/usr/local/spark-1.1.0-bin-hadoop2.3/work/app-20150414185611-0000/0/./ant-launcher-1.8.2.jar to class loader
15/04/14 18:56:13 INFO Executor: Fetching http://172.17.0.16:45360/jars/adam-apis-0.15.0.jar with timestamp 1429037770792
15/04/14 18:56:13 INFO Utils: Fetching http://172.17.0.16:45360/jars/adam-apis-0.15.0.jar to /tmp/fetchFileTemp452417511092631568.tmp
15/04/14 18:56:13 INFO Executor: Adding file:/usr/local/spark-1.1.0-bin-hadoop2.3/work/app-20150414185611-0000/0/./adam-apis-0.15.0.jar to class loader
15/04/14 18:56:13 INFO Executor: Fetching http://172.17.0.16:45360/jars/guava-14.0.1.jar with timestamp 1429037770696
15/04/14 18:56:13 INFO Utils: Fetching http://172.17.0.16:45360/jars/guava-14.0.1.jar to /tmp/fetchFileTemp5078180267466695914.tmp
15/04/14 18:56:13 INFO Executor: Adding file:/usr/local/spark-1.1.0-bin-hadoop2.3/work/app-20150414185611-0000/0/./guava-14.0.1.jar to class loader
15/04/14 18:56:13 INFO Executor: Fetching http://172.17.0.16:45360/jars/reflectasm-1.07-shaded.jar with timestamp 1429037770719
15/04/14 18:56:13 INFO Utils: Fetching http://172.17.0.16:45360/jars/reflectasm-1.07-shaded.jar to /tmp/fetchFileTemp1888943180233620703.tmp
15/04/14 18:56:13 INFO Executor: Adding file:/usr/local/spark-1.1.0-bin-hadoop2.3/work/app-20150414185611-0000/0/./reflectasm-1.07-shaded.jar to class loader
15/04/14 18:56:13 INFO Executor: Fetching http://172.17.0.16:45360/jars/bdg-formats-0.4.0.jar with timestamp 1429037770705
15/04/14 18:56:13 INFO Utils: Fetching http://172.17.0.16:45360/jars/bdg-formats-0.4.0.jar to /tmp/fetchFileTemp7247129694857228863.tmp
15/04/14 18:56:13 INFO Executor: Adding file:/usr/local/spark-1.1.0-bin-hadoop2.3/work/app-20150414185611-0000/0/./bdg-formats-0.4.0.jar to class loader
15/04/14 18:56:13 INFO Executor: Fetching http://172.17.0.16:45360/jars/json4s-core_2.10-3.2.10.jar with timestamp 1429037770848
15/04/14 18:56:13 INFO Utils: Fetching http://172.17.0.16:45360/jars/json4s-core_2.10-3.2.10.jar to /tmp/fetchFileTemp5003181348880161547.tmp
15/04/14 18:56:13 INFO Executor: Adding file:/usr/local/spark-1.1.0-bin-hadoop2.3/work/app-20150414185611-0000/0/./json4s-core_2.10-3.2.10.jar to class loader
15/04/14 18:56:13 INFO Executor: Fetching http://172.17.0.16:45360/jars/juniversalchardet-1.0.3.jar with timestamp 1429037770858
15/04/14 18:56:13 INFO Utils: Fetching http://172.17.0.16:45360/jars/juniversalchardet-1.0.3.jar to /tmp/fetchFileTemp3163585313980219109.tmp
15/04/14 18:56:13 INFO Executor: Adding file:/usr/local/spark-1.1.0-bin-hadoop2.3/work/app-20150414185611-0000/0/./juniversalchardet-1.0.3.jar to class loader
15/04/14 18:56:13 INFO Executor: Fetching http://172.17.0.16:45360/jars/scalatra_2.10-2.3.0.jar with timestamp 1429037770855
15/04/14 18:56:13 INFO Utils: Fetching http://172.17.0.16:45360/jars/scalatra_2.10-2.3.0.jar to /tmp/fetchFileTemp8229749564150709625.tmp
15/04/14 18:56:13 INFO Executor: Adding file:/usr/local/spark-1.1.0-bin-hadoop2.3/work/app-20150414185611-0000/0/./scalatra_2.10-2.3.0.jar to class loader
15/04/14 18:56:13 INFO Executor: Fetching http://172.17.0.16:45360/jars/scalatra-common_2.10-2.3.0.jar with timestamp 1429037770856
15/04/14 18:56:13 INFO Utils: Fetching http://172.17.0.16:45360/jars/scalatra-common_2.10-2.3.0.jar to /tmp/fetchFileTemp7222374969558899826.tmp
15/04/14 18:56:13 INFO Executor: Adding file:/usr/local/spark-1.1.0-bin-hadoop2.3/work/app-20150414185611-0000/0/./scalatra-common_2.10-2.3.0.jar to class loader
15/04/14 18:56:13 INFO Executor: Fetching http://172.17.0.16:45360/jars/parquet-encoding-1.6.0rc4.jar with timestamp 1429037770760
15/04/14 18:56:13 INFO Utils: Fetching http://172.17.0.16:45360/jars/parquet-encoding-1.6.0rc4.jar to /tmp/fetchFileTemp3713177571411689667.tmp
15/04/14 18:56:13 INFO Executor: Adding file:/usr/local/spark-1.1.0-bin-hadoop2.3/work/app-20150414185611-0000/0/./parquet-encoding-1.6.0rc4.jar to class loader
15/04/14 18:56:13 INFO Executor: Fetching http://172.17.0.16:45360/jars/annotations-2.0.0.jar with timestamp 1429037770792
15/04/14 18:56:13 INFO Utils: Fetching http://172.17.0.16:45360/jars/annotations-2.0.0.jar to /tmp/fetchFileTemp2076170929340010228.tmp
15/04/14 18:56:13 INFO Executor: Adding file:/usr/local/spark-1.1.0-bin-hadoop2.3/work/app-20150414185611-0000/0/./annotations-2.0.0.jar to class loader
15/04/14 18:56:13 INFO Executor: Fetching http://172.17.0.16:45360/jars/parquet-column-1.6.0rc4.jar with timestamp 1429037770758
15/04/14 18:56:13 INFO Utils: Fetching http://172.17.0.16:45360/jars/parquet-column-1.6.0rc4.jar to /tmp/fetchFileTemp2383576209123461404.tmp
15/04/14 18:56:13 INFO Executor: Adding file:/usr/local/spark-1.1.0-bin-hadoop2.3/work/app-20150414185611-0000/0/./parquet-column-1.6.0rc4.jar to class loader
15/04/14 18:56:13 INFO Executor: Fetching http://172.17.0.16:45360/jars/scalate-util_2.10-1.6.1.jar with timestamp 1429037770816
15/04/14 18:56:13 INFO Utils: Fetching http://172.17.0.16:45360/jars/scalate-util_2.10-1.6.1.jar to /tmp/fetchFileTemp1285303994951399648.tmp
15/04/14 18:56:13 INFO Executor: Adding file:/usr/local/spark-1.1.0-bin-hadoop2.3/work/app-20150414185611-0000/0/./scalate-util_2.10-1.6.1.jar to class loader
15/04/14 18:56:13 INFO TorrentBroadcast: Started reading broadcast variable 1
15/04/14 18:56:13 INFO SendingConnection: Initiating connection to [adam/172.17.0.16:52263]
15/04/14 18:56:13 INFO SendingConnection: Connected to [adam/172.17.0.16:52263], 1 messages pending
15/04/14 18:56:33 INFO ConnectionManager: Accepted connection from [adam/172.17.0.16:34797]
15/04/14 18:56:33 INFO MemoryStore: ensureFreeSpace(22331) called with curMem=0, maxMem=1111794647
15/04/14 18:56:33 INFO MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 21.8 KB, free 1060.3 MB)
15/04/14 18:56:33 INFO BlockManagerMaster: Updated info of block broadcast_1_piece0
15/04/14 18:56:33 INFO TorrentBroadcast: Reading broadcast variable 1 took 20.130715702 s
15/04/14 18:56:34 INFO MemoryStore: ensureFreeSpace(64336) called with curMem=22331, maxMem=1111794647
15/04/14 18:56:34 INFO MemoryStore: Block broadcast_1 stored as values in memory (estimated size 62.8 KB, free 1060.2 MB)
15/04/14 18:56:34 INFO TorrentBroadcast: Started reading broadcast variable 0
15/04/14 18:56:44 INFO MemoryStore: ensureFreeSpace(15731) called with curMem=86667, maxMem=1111794647
15/04/14 18:56:44 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 15.4 KB, free 1060.2 MB)
15/04/14 18:56:44 INFO BlockManagerMaster: Updated info of block broadcast_0_piece0
15/04/14 18:56:44 INFO TorrentBroadcast: Reading broadcast variable 0 took 10.02929592 s
15/04/14 18:56:44 INFO MemoryStore: ensureFreeSpace(264748) called with curMem=102398, maxMem=1111794647
15/04/14 18:56:44 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 258.5 KB, free 1059.9 MB)
15/04/14 18:56:44 INFO deprecation: mapred.output.dir is deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir
15/04/14 18:56:44 INFO NewHadoopRDD: Input split: hdfs://hdfs-namenode:9000/SRR062634.sam:0+94792807
15/04/14 18:56:45 INFO CodecPool: Got brand-new compressor [.gz]
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
15/04/14 18:57:00 INFO FileOutputCommitter: Saved output of task 'attempt_201504141856_0002_r_000000_0' to hdfs://hdfs-namenode:9000/SRR062634b.adam/_temporary/0/task_201504141856_0002_r_000000
15/04/14 18:57:00 INFO Executor: Finished task 0.0 in stage 0.0 (TID 0). 1846 bytes result sent to driver
15/04/14 18:57:01 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(adam,52263)
15/04/14 18:57:01 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(adam,52263)
15/04/14 18:57:01 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(adam,52263)
15/04/14 18:57:01 INFO ConnectionManager: key already cancelled ? sun.nio.ch.SelectionKeyImpl@2662e5cf
java.nio.channels.CancelledKeyException
at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:73)
at sun.nio.ch.SelectionKeyImpl.readyOps(SelectionKeyImpl.java:87)
at java.nio.channels.SelectionKey.isConnectable(SelectionKey.java:336)
at org.apache.spark.network.ConnectionManager.run(ConnectionManager.scala:375)
at org.apache.spark.network.ConnectionManager$$anon$4.run(ConnectionManager.scala:139)
15/04/14 18:57:01 ERROR CoarseGrainedExecutorBackend: Driver Disassociated [akka.tcp://sparkExecutor@spark-worker1:37876] -> [akka.tcp://sparkDriver@adam:33997] disassociated! Shutting down.
@davidonlaptop @flangelier As you are using hostnames, have you try in hdfs-site.xml to set the property dfs.namenode.datanode.registration.ip-hostname-check to true instead of false?
Consequence
Worker is listed in the removed executors list with its state as KILLED.
However this is an output file being produced, it is not clear yet if this is a real problem.
Steps to reproduce
STDERR
STDOUT