search

swsnu / bdcsfall2014

0 stars 0 forks source link

Having problems with running programs Local mode and on YARN #24

Open kim333 opened 9 years ago

kim333 commented 9 years ago

We are trying to run our program with REEF and we ran into problems with the YARN cluster. It runs fine when we run the program with local mode but when we run it on the YARN, we sometimes have problems such as "Cannot append by read-append-delete-create with exception" or "Cannot open or write to log file".

image

Sometimes it says succeeded but when we look at the log files, it doesn't print out anything. But as we said earlier, when we run it on local mode, it does print out logs

bchocho commented 9 years ago

@kim333 Can you attach the part of the Driver or Task log that produces Cannot append by read-append-delete-create with exception.?

hanmanhui commented 9 years ago 
2014-11-17 12:44:46,749 FINEST reef.wake.time.runtime.RuntimeClock.run main | Entering clock main loop iteration.
2014-11-17 12:44:46,797 FINEST hadoop.ipc.ProtobufRpcEngine.invoke AMRM Callback Handler Thread | 47: Call -> cluster-01.ids.snu.ac.kr/147.46.215.66:8022: finishApplicationMaster {diagnostics: "Cannot append by read-append-delete-create with exception." final_application_status: APP_FAILED}
2014-11-17 12:44:46,798 FINE hadoop.ipc.Client.run IPC Parameter Sending Thread #1 | IPC Client (1007777944) connection to cluster-01.ids.snu.ac.kr/147.46.215.66:8022 from hadoop sending #63
2014-11-17 12:44:46,798 FINE hadoop.ipc.Client.receiveRpcResponse IPC Client (1007777944) connection to cluster-01.ids.snu.ac.kr/147.46.215.66:8022 from hadoop | IPC Client (1007777944) connection to cluster-01.ids.snu.ac.kr/147.46.215.66:8022 from hadoop got value #63
2014-11-17 12:44:46,799 FINE hadoop.ipc.ProtobufRpcEngine.invoke AMRM Callback Handler Thread | Call: finishApplicationMaster took 2ms
2014-11-17 12:44:46,799 FINEST hadoop.ipc.ProtobufRpcEngine.invoke AMRM Callback Handler Thread | 47: Response <- cluster-01.ids.snu.ac.kr/147.46.215.66:8022: finishApplicationMaster {isUnregistered: true}
2014-11-17 12:44:46,799 FINE hadoop.service.AbstractService.enterState AMRM Callback Handler Thread | Service: org.apache.hadoop.yarn.client.api.async.AMRMClientAsync entered state STOPPED
2014-11-17 12:44:46,799 FINE hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl.run AMRM Heartbeater thread | Heartbeater interrupted
java.lang.InterruptedException: sleep interrupted
    at java.lang.Thread.sleep(Native Method)
    at org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$HeartbeatThread.run(AMRMClientAsyncImpl.java:249)

2014-11-17 12:44:46,800 FINE hadoop.service.AbstractService.enterState AMRM Callback Handler Thread | Service: org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl entered state STOPPED
2014-11-17 12:44:46,800 FINE hadoop.ipc.Client.stopClient AMRM Callback Handler Thread | stopping client from cache: org.apache.hadoop.ipc.Client@3fde1ed4
2014-11-17 12:44:46,821 FINEST reef.runtime.common.driver.resourcemanager.ResourceManagerStatus.onNext AMRM Callback Handler Thread | Runtime status name: "YARN"
state: FAILED
error {
  name: "YARN"
  message: "Cannot append by read-append-delete-create with exception."
  exception: "\254\355\000\005sr\000\032java.lang.RuntimeException\236_\006G\n4\203\345\002\000\000xr\000\023java.lang.Exception\320\375\037>\032;\034\304\002\000\000xr\000\023java.lang.Throwable\325\3065\'9w\270\313\003\000\004L\000\005causet\000\025Ljava/lang/Throwable;L\000\rdetailMessaget\000\022Ljava/lang/String;[\000\nstackTracet\000\036[Ljava/lang/StackTraceElement;L\000\024suppressedExceptionst\000\020Ljava/util/List;xpsr\000\035java.io.FileNotFoundException\363\212+\326\340!j\344\002\000\000xr\000\023java.io.IOExceptionl\200sde%\360\253\002\000\000xq\000~\000\001sr\000%org.apache.hadoop.ipc.RemoteException\000\000\000\000\000\000\000\001\002\000\002I\000\terrorCodeL\000\tclassNameq\000~\000\004xq\000~\000\tq\000~\000\ft\006]File does not exist: /ReefApplications/application_1415768120886_0029/evaluatorsChangesLog\n\tat org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:65)\n\tat org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:55)\n\tat org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsUpdateTimes(FSNamesystem.java:1716)\n\tat org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:1659)\n\tat org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1639)\n\tat org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1613)\n\tat org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:497)\n\tat org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:322)\n\tat org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)\n\tat org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)\n\tat org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928)\n\tat org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2013)\n\tat org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2009)\n\tat java.security.AccessController.doPrivileged(Native Method)\n\tat javax.security.auth.Subject.doAs(Subject.java:415)\n\tat org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)\n\tat org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)\nur\000\036[Ljava.lang.StackTraceElement;\002F*<<\375\"9\002\000\000xp\000\000\000\036sr\000\033java.lang.StackTraceElementa\t\305\232&6\335\205\002\000\004I\000\nlineNumberL\000\016declaringClassq\000~\000\004L\000\bfileNameq\000~\000\004L\000\nmethodNameq\000~\000\004xp\000\000\005\202t\000\034org.apache.hadoop.ipc.Clientt\000\vClient.javat\000\004callsq\000~\000\020\000\000\005Sq\000~\000\022q\000~\000\023q\000~\000\024sq\000~\000\020\000\000\000\316t\000/org.apache.hadoop.ipc.ProtobufRpcEngine$Invokert\000\026ProtobufRpcEngine.javat\000\006invokesq\000~\000\020\377\377\377\377t\000\026com.sun.proxy.$Proxy25pt\000\021getBlockLocationssq\000~\000\020\377\377\377\376t\000$sun.reflect.NativeMethodAccessorImplt\000\035NativeMethodAccessorImpl.javat\000\ainvoke0sq\000~\000\020\000\000\0009q\000~\000\036q\000~\000\037q\000~\000\031sq\000~\000\020\000\000\000+t\000(sun.reflect.DelegatingMethodAccessorImplt\000!DelegatingMethodAccessorImpl.javaq\000~\000\031sq\000~\000\020\000\000\002^t\000\030java.lang.reflect.Methodt\000\vMethod.javaq\000~\000\031sq\000~\000\020\000\000\000\276t\0001org.apache.hadoop.io.retry.RetryInvocationHandlert\000\033RetryInvocationHandler.javat\000\finvokeMethodsq\000~\000\020\000\000\000gq\000~\000)q\000~\000*q\000~\000\031sq\000~\000\020\377\377\377\377q\000~\000\033pq\000~\000\034sq\000~\000\020\000\000\000\333t\000Dorg.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPBt\000\'ClientNamenodeProtocolTranslatorPB.javaq\000~\000\034sq\000~\000\020\000\000\004vt\000 org.apache.hadoop.hdfs.DFSClientt\000\016DFSClient.javat\000\025callGetBlockLocationssq\000~\000\020\000\000\004lq\000~\0002q\000~\0003t\000\020getLocatedBlockssq\000~\000\020\000\000\004bq\000~\0002q\000~\0003q\000~\0006sq\000~\000\020\000\000\001\bt\000%org.apache.hadoop.hdfs.DFSInputStreamt\000\023DFSInputStream.javat\000\'fetchLocatedBlocksAndGetLastBlockLengthsq\000~\000\020\000\000\000\347q\000~\0009q\000~\000:t\000\bopenInfosq\000~\000\020\000\000\000\340q\000~\0009q\000~\000:t\000\006<init>sq\000~\000\020\000\000\005\017q\000~\0002q\000~\0003t\000\004opensq\000~\000\020\000\000\001,t\000.org.apache.hadoop.hdfs.DistributedFileSystem$3t\000\032DistributedFileSystem.javat\000\006doCallsq\000~\000\020\000\000\001(q\000~\000Cq\000~\000Dq\000~\000Esq\000~\000\020\000\000\000Qt\000+org.apache.hadoop.fs.FileSystemLinkResolvert\000\033FileSystemLinkResolver.javat\000\aresolvesq\000~\000\020\000\000\001(t\000,org.apache.hadoop.hdfs.DistributedFileSystemq\000~\000Dq\000~\000Asq\000~\000\020\000\000\002\374t\000\037org.apache.hadoop.fs.FileSystemt\000\017FileSystem.javaq\000~\000Asq\000~\000\020\000\000\002ft\0008org.apache.reef.runtime.yarn.driver.YarnContainerManagert\000\031YarnContainerManager.javat\000\027appendByDeleteAndCreatesq\000~\000\020\000\000\002Wq\000~\000Qq\000~\000Rt\000\023writeToEvaluatorLogsq\000~\000\020\000\000\002{q\000~\000Qq\000~\000Rt\000\024logContainerAdditionsq\000~\000\020\000\000\001\317q\000~\000Qq\000~\000Rt\000\022handleNewContainersq\000~\000\020\000\000\000\204q\000~\000Qq\000~\000Rt\000\025onContainersAllocatedsq\000~\000\020\000\000\0014t\000Vorg.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$CallbackHandlerThreadt\000\030AMRMClientAsyncImpl.javat\000\003runsr\000&java.util.Collections$UnmodifiableList\374\017%1\265\354\216\020\002\000\001L\000\004listq\000~\000\006xr\000,java.util.Collections$UnmodifiableCollection\031B\000\200\313^\367\036\002\000\001L\000\001ct\000\026Ljava/util/Collection;xpsr\000\023java.util.ArrayListx\201\322\035\231\307a\235\003\000\001I\000\004sizexp\000\000\000\000w\004\000\000\000\000xq\000~\000ex\000\000\000\001t\000\035java.io.FileNotFoundExceptionq\000~\000\ruq\000~\000\016\000\000\000\030sq\000~\000\020\377\377\377\376t\000)sun.reflect.NativeConstructorAccessorImplt\000\"NativeConstructorAccessorImpl.javat\000\fnewInstance0sq\000~\000\020\000\000\0009q\000~\000iq\000~\000jt\000\vnewInstancesq\000~\000\020\000\000\000-t\000-sun.reflect.DelegatingConstructorAccessorImplt\000&DelegatingConstructorAccessorImpl.javaq\000~\000msq\000~\000\020\000\000\002\016t\000\035java.lang.reflect.Constructort\000\020Constructor.javaq\000~\000msq\000~\000\020\000\000\000jt\000%org.apache.hadoop.ipc.RemoteExceptiont\000\024RemoteException.javat\000\024instantiateExceptionsq\000~\000\020\000\000\000Iq\000~\000uq\000~\000vt\000\025unwrapRemoteExceptionsq\000~\000\020\000\000\004xq\000~\0002q\000~\0003q\000~\0004sq\000~\000\020\000\000\004lq\000~\0002q\000~\0003q\000~\0006sq\000~\000\020\000\000\004bq\000~\0002q\000~\0003q\000~\0006sq\000~\000\020\000\000\001\bq\000~\0009q\000~\000:q\000~\000;sq\000~\000\020\000\000\000\347q\000~\0009q\000~\000:q\000~\000=sq\000~\000\020\000\000\000\340q\000~\0009q\000~\000:q\000~\000?sq\000~\000\020\000\000\005\017q\000~\0002q\000~\0003q\000~\000Asq\000~\000\020\000\000\001,q\000~\000Cq\000~\000Dq\000~\000Esq\000~\000\020\000\000\001(q\000~\000Cq\000~\000Dq\000~\000Esq\000~\000\020\000\000\000Qq\000~\000Hq\000~\000Iq\000~\000Jsq\000~\000\020\000\000\001(q\000~\000Lq\000~\000Dq\000~\000Asq\000~\000\020\000\000\002\374q\000~\000Nq\000~\000Oq\000~\000Asq\000~\000\020\000\000\002fq\000~\000Qq\000~\000Rq\000~\000Ssq\000~\000\020\000\000\002Wq\000~\000Qq\000~\000Rq\000~\000Usq\000~\000\020\000\000\002{q\000~\000Qq\000~\000Rq\000~\000Wsq\000~\000\020\000\000\001\317q\000~\000Qq\000~\000Rq\000~\000Ysq\000~\000\020\000\000\000\204q\000~\000Qq\000~\000Rq\000~\000[sq\000~\000\020\000\000\0014q\000~\000]q\000~\000^q\000~\000_q\000~\000cxt\000:Cannot append by read-append-delete-create with exception.uq\000~\000\016\000\000\000\006sq\000~\000\020\000\000\002qq\000~\000Qq\000~\000Rq\000~\000Ssq\000~\000\020\000\000\002Wq\000~\000Qq\000~\000Rq\000~\000Usq\000~\000\020\000\000\002{q\000~\000Qq\000~\000Rq\000~\000Wsq\000~\000\020\000\000\001\317q\000~\000Qq\000~\000Rq\000~\000Ysq\000~\000\020\000\000\000\204q\000~\000Qq\000~\000Rq\000~\000[sq\000~\000\020\000\000\0014q\000~\000]q\000~\000^q\000~\000_q\000~\000cx"
}

2014-11-17 12:44:46,822 FINER reef.runtime.common.driver.DriverStatusManager.onError AMRM Callback Handler Thread | ENTRY java.lang.Exception: Resource Manager failure
2014-11-17 12:44:46,822 WARNING reef.runtime.common.driver.DriverStatusManager.onError AMRM Callback Handler Thread | Shutting down the Driver with an exception: 
java.lang.Exception: Resource Manager failure
    at org.apache.reef.runtime.common.driver.resourcemanager.ResourceManagerErrorHandler.onNext(ResourceManagerErrorHandler.java:42)
    at org.apache.reef.runtime.common.driver.resourcemanager.ResourceManagerStatus.onRMFailure(ResourceManagerStatus.java:115)
    at org.apache.reef.runtime.common.driver.resourcemanager.ResourceManagerStatus.onNext(ResourceManagerStatus.java:76)
    at org.apache.reef.runtime.common.driver.resourcemanager.ResourceManagerStatus.onNext(ResourceManagerStatus.java:39)
    at org.apache.reef.runtime.yarn.driver.REEFEventHandlers.onRuntimeStatus(REEFEventHandlers.java:66)
    at org.apache.reef.runtime.yarn.driver.YarnContainerManager.onRuntimeError(YarnContainerManager.java:521)
    at org.apache.reef.runtime.yarn.driver.YarnContainerManager.onError(YarnContainerManager.java:159)
    at org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$CallbackHandlerThread.run(AMRMClientAsyncImpl.java:313)

2014-11-17 12:44:46,822 FINER reef.wake.time.runtime.RuntimeClock.stop AMRM Callback Handler Thread | ENTRY
2014-11-17 12:44:46,822 FINER reef.wake.time.runtime.RuntimeClock.stop AMRM Callback Handler Thread | RETURN
2014-11-17 12:44:46,822 FINEST reef.wake.impl.PubSubEventHandler.onNext main | Invoked for event: org.apache.reef.wake.time.event.StopTime[1416195886822]
2014-11-17 12:44:46,822 FINER reef.runtime.common.driver.DriverStatusManager.onError AMRM Callback Handler Thread | RETURN [Ljava.lang.Object;@270fef98
2014-11-17 12:44:46,823 FINEST reef.wake.impl.PubSubEventHandler.onNext main | Invoking org.apache.reef.wake.impl.LoggingEventHandler@33914003
2014-11-17 12:44:46,823 SEVERE reef.runtime.common.launch.REEFUncaughtExceptionHandler.uncaughtException AMRM Callback Handler Thread | Thread AMRM Callback Handler Thread threw an uncaught exception.
org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.lang.RuntimeException: Cannot append by read-append-delete-create with exception.
    at org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$CallbackHandlerThread.run(AMRMClientAsyncImpl.java:315)
Caused by: java.lang.RuntimeException: Cannot append by read-append-delete-create with exception.
    at org.apache.reef.runtime.yarn.driver.YarnContainerManager.appendByDeleteAndCreate(YarnContainerManager.java:625)
    at org.apache.reef.runtime.yarn.driver.YarnContainerManager.writeToEvaluatorLog(YarnContainerManager.java:599)
    at org.apache.reef.runtime.yarn.driver.YarnContainerManager.logContainerAddition(YarnContainerManager.java:635)
    at org.apache.reef.runtime.yarn.driver.YarnContainerManager.handleNewContainer(YarnContainerManager.java:463)
    at org.apache.reef.runtime.yarn.driver.YarnContainerManager.onContainersAllocated(YarnContainerManager.java:132)
    at org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$CallbackHandlerThread.run(AMRMClientAsyncImpl.java:308)
Caused by: java.io.FileNotFoundException: File does not exist: /ReefApplications/application_1415768120886_0029/evaluatorsChangesLog
    at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:65)
    at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:55)
    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsUpdateTimes(FSNamesystem.java:1716)
    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:1659)
    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1639)
    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1613)
    at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:497)
    at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:322)
    at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
    at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
    at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2013)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2009)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)

    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
    at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
    at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:73)
    at org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:1144)
    at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1132)
    at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1122)
    at org.apache.hadoop.hdfs.DFSInputStream.fetchLocatedBlocksAndGetLastBlockLength(DFSInputStream.java:264)
    at org.apache.hadoop.hdfs.DFSInputStream.openInfo(DFSInputStream.java:231)
    at org.apache.hadoop.hdfs.DFSInputStream.<init>(DFSInputStream.java:224)
    at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:1295)
    at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:300)
    at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:296)
    at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
    at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:296)
    at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:764)
    at org.apache.reef.runtime.yarn.driver.YarnContainerManager.appendByDeleteAndCreate(YarnContainerManager.java:614)
    ... 5 more
Caused by: org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): File does not exist: /ReefApplications/application_1415768120886_0029/evaluatorsChangesLog
    at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:65)
    at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:55)
    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsUpdateTimes(FSNamesystem.java:1716)
    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:1659)
    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1639)
    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1613)
    at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:497)
    at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:322)
    at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
    at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
    at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2013)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2009)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)

    at org.apache.hadoop.ipc.Client.call(Client.java:1410)
    at org.apache.hadoop.ipc.Client.call(Client.java:1363)
    at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
    at com.sun.proxy.$Proxy25.getBlockLocations(Unknown Source)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:190)
    at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:103)
    at com.sun.proxy.$Proxy25.getBlockLocations(Unknown Source)
    at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getBlockLocations(ClientNamenodeProtocolTranslatorPB.java:219)
    at org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:1142)
    ... 17 more

2014-11-17 12:44:46,823 FINE reef.wake.impl.LoggingEventHandler.onNext main | org.apache.reef.wake.time.event.StopTime[1416195886822]
2014-11-17 12:44:46,823 SEVERE reef.runtime.common.launch.REEFErrorHandler.onNext AMRM Callback Handler Thread | Uncaught exception.
java.lang.Exception: Thread AMRM Callback Handler Thread threw an uncaught exception.
    at org.apache.reef.runtime.common.launch.REEFUncaughtExceptionHandler.uncaughtException(REEFUncaughtExceptionHandler.java:50)
    at java.lang.ThreadGroup.uncaughtException(ThreadGroup.java:1057)
    at java.lang.ThreadGroup.uncaughtException(ThreadGroup.java:1052)
    at java.lang.Thread.dispatchUncaughtException(Thread.java:1965)
Caused by: org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.lang.RuntimeException: Cannot append by read-append-delete-create with exception.
    at org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$CallbackHandlerThread.run(AMRMClientAsyncImpl.java:315)
Caused by: java.lang.RuntimeException: Cannot append by read-append-delete-create with exception.
    at org.apache.reef.runtime.yarn.driver.YarnContainerManager.appendByDeleteAndCreate(YarnContainerManager.java:625)
    at org.apache.reef.runtime.yarn.driver.YarnContainerManager.writeToEvaluatorLog(YarnContainerManager.java:599)
    at org.apache.reef.runtime.yarn.driver.YarnContainerManager.logContainerAddition(YarnContainerManager.java:635)
    at org.apache.reef.runtime.yarn.driver.YarnContainerManager.handleNewContainer(YarnContainerManager.java:463)
    at org.apache.reef.runtime.yarn.driver.YarnContainerManager.onContainersAllocated(YarnContainerManager.java:132)
    at org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$CallbackHandlerThread.run(AMRMClientAsyncImpl.java:308)
Caused by: java.io.FileNotFoundException: File does not exist: /ReefApplications/application_1415768120886_0029/evaluatorsChangesLog
    at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:65)
    at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:55)
    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsUpdateTimes(FSNamesystem.java:1716)
    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:1659)
    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1639)
    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1613)
    at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:497)
    at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:322)
    at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
    at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
    at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2013)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2009)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)

    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
    at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
    at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:73)
    at org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:1144)
    at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1132)
    at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1122)
    at org.apache.hadoop.hdfs.DFSInputStream.fetchLocatedBlocksAndGetLastBlockLength(DFSInputStream.java:264)
    at org.apache.hadoop.hdfs.DFSInputStream.openInfo(DFSInputStream.java:231)
    at org.apache.hadoop.hdfs.DFSInputStream.<init>(DFSInputStream.java:224)
    at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:1295)
    at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:300)
    at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:296)
    at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
    at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:296)
    at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:764)
    at org.apache.reef.runtime.yarn.driver.YarnContainerManager.appendByDeleteAndCreate(YarnContainerManager.java:614)
    ... 5 more
Caused by: org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): File does not exist: /ReefApplications/application_1415768120886_0029/evaluatorsChangesLog
    at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:65)
    at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:55)
    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsUpdateTimes(FSNamesystem.java:1716)
    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:1659)
    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1639)
    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1613)
    at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:497)
    at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:322)
    at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
    at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
    at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2013)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2009)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)

    at org.apache.hadoop.ipc.Client.call(Client.java:1410)
    at org.apache.hadoop.ipc.Client.call(Client.java:1363)
    at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
    at com.sun.proxy.$Proxy25.getBlockLocations(Unknown Source)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:190)
    at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:103)
    at com.sun.proxy.$Proxy25.getBlockLocations(Unknown Source)
    at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getBlockLocations(ClientNamenodeProtocolTranslatorPB.java:219)
    at org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:1142)
    ... 17 more

2014-11-17 12:44:46,824 FINEST reef.wake.impl.PubSubEventHandler.onNext main | Invoked for event: org.apache.reef.wake.time.runtime.event.RuntimeStop[1416195886823]
2014-11-17 12:44:46,824 FINEST reef.wake.impl.PubSubEventHandler.onNext main | Invoking org.apache.reef.runtime.yarn.driver.YARNRuntimeStopHandler@402425cb
2014-11-17 12:44:46,824 FINE reef.wake.remote.impl.DefaultRemoteManagerImplementation.getHandler AMRM Callback Handler Thread | RemoteManager: REEF_LAUNCHER destinationIdentifier: socket://147.46.215.66:11066 messageType: org.apache.reef.proto.ReefServiceProtos$RuntimeErrorProto
2014-11-17 12:44:46,824 FINE reef.runtime.yarn.driver.YarnContainerManager.onStop main | Stop Runtime: RM status STOPPED
2014-11-17 12:44:46,824 FINE reef.wake.remote.impl.ProxyEventHandler.<init> AMRM Callback Handler Thread | ProxyEventHandler myId: socket://147.46.215.66:14140 remoteId: socket://147.46.215.66:11066 remoteSink: default handler: org.apache.reef.wake.remote.impl.RemoteSenderEventHandler@34a83912
2014-11-17 12:44:46,825 FINE hadoop.service.AbstractService.enterState main | Service: org.apache.hadoop.yarn.client.api.async.NMClientAsync entered state STOPPED
2014-11-17 12:44:46,826 FINE hadoop.service.AbstractService.enterState main | Service: org.apache.hadoop.yarn.client.api.impl.NMClientImpl entered state STOPPED
2014-11-17 12:44:46,828 FINE reef.wake.remote.impl.ProxyEventHandler.onNext AMRM Callback Handler Thread | remoteid: /147.46.215.66:11066
name: "reef"
message: "Thread AMRM Callback Handler Thread threw an uncaught exception."
exception: "\254\355\000\005sr\000\023java.lang.Exception\320\375\037>\032;\034\304\002\000\000xr\000\023java.lang.Throwable\325\3065\'9w\270\313\003\000\004L\000\005causet\000\025Ljava/lang/Throwable;L\000\rdetailMessaget\000\022Ljava/lang/String;[\000\nstackTracet\000\036[Ljava/lang/StackTraceElement;L\000\024suppressedExceptionst\000\020Ljava/util/List;xpsr\0006org.apache.hadoop.yarn.exceptions.YarnRuntimeException\234\272\356\334\361\207\212`\002\000\000xr\000\032java.lang.RuntimeException\236_\006G\n4\203\345\002\000\000xq\000~\000\000sq\000~\000\bsr\000\035java.io.FileNotFoundException\363\212+\326\340!j\344\002\000\000xr\000\023java.io.IOExceptionl\200sde%\360\253\002\000\000xq\000~\000\000sr\000%org.apache.hadoop.ipc.RemoteException\000\000\000\000\000\000\000\001\002\000\002I\000\terrorCodeL\000\tclassNameq\000~\000\003xq\000~\000\fq\000~\000\017t\006]File does not exist: /ReefApplications/application_1415768120886_0029/evaluatorsChangesLog\n\tat org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:65)\n\tat org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:55)\n\tat org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsUpdateTimes(FSNamesystem.java:1716)\n\tat org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:1659)\n\tat org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1639)\n\tat org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1613)\n\tat org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:497)\n\tat org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:322)\n\tat org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)\n\tat org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)\n\tat org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928)\n\tat org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2013)\n\tat org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2009)\n\tat java.security.AccessController.doPrivileged(Native Method)\n\tat javax.security.auth.Subject.doAs(Subject.java:415)\n\tat org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)\n\tat org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)\nur\000\036[Ljava.lang.StackTraceElement;\002F*<<\375\"9\002\000\000xp\000\000\000\036sr\000\033java.lang.StackTraceElementa\t\305\232&6\335\205\002\000\004I\000\nlineNumberL\000\016declaringClassq\000~\000\003L\000\bfileNameq\000~\000\003L\000\nmethodNameq\000~\000\003xp\000\000\005\202t\000\034org.apache.hadoop.ipc.Clientt\000\vClient.javat\000\004callsq\000~\000\023\000\000\005Sq\000~\000\025q\000~\000\026q\000~\000\027sq\000~\000\023\000\000\000\316t\000/org.apache.hadoop.ipc.ProtobufRpcEngine$Invokert\000\026ProtobufRpcEngine.javat\000\006invokesq\000~\000\023\377\377\377\377t\000\026com.sun.proxy.$Proxy25pt\000\021getBlockLocationssq\000~\000\023\377\377\377\376t\000$sun.reflect.NativeMethodAccessorImplt\000\035NativeMethodAccessorImpl.javat\000\ainvoke0sq\000~\000\023\000\000\0009q\000~\000!q\000~\000\"q\000~\000\034sq\000~\000\023\000\000\000+t\000(sun.reflect.DelegatingMethodAccessorImplt\000!DelegatingMethodAccessorImpl.javaq\000~\000\034sq\000~\000\023\000\000\002^t\000\030java.lang.reflect.Methodt\000\vMethod.javaq\000~\000\034sq\000~\000\023\000\000\000\276t\0001org.apache.hadoop.io.retry.RetryInvocationHandlert\000\033RetryInvocationHandler.javat\000\finvokeMethodsq\000~\000\023\000\000\000gq\000~\000,q\000~\000-q\000~\000\034sq\000~\000\023\377\377\377\377q\000~\000\036pq\000~\000\037sq\000~\000\023\000\000\000\333t\000Dorg.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPBt\000\'ClientNamenodeProtocolTranslatorPB.javaq\000~\000\037sq\000~\000\023\000\000\004vt\000 org.apache.hadoop.hdfs.DFSClientt\000\016DFSClient.javat\000\025callGetBlockLocationssq\000~\000\023\000\000\004lq\000~\0005q\000~\0006t\000\020getLocatedBlockssq\000~\000\023\000\000\004bq\000~\0005q\000~\0006q\000~\0009sq\000~\000\023\000\000\001\bt\000%org.apache.hadoop.hdfs.DFSInputStreamt\000\023DFSInputStream.javat\000\'fetchLocatedBlocksAndGetLastBlockLengthsq\000~\000\023\000\000\000\347q\000~\000<q\000~\000=t\000\bopenInfosq\000~\000\023\000\000\000\340q\000~\000<q\000~\000=t\000\006<init>sq\000~\000\023\000\000\005\017q\000~\0005q\000~\0006t\000\004opensq\000~\000\023\000\000\001,t\000.org.apache.hadoop.hdfs.DistributedFileSystem$3t\000\032DistributedFileSystem.javat\000\006doCallsq\000~\000\023\000\000\001(q\000~\000Fq\000~\000Gq\000~\000Hsq\000~\000\023\000\000\000Qt\000+org.apache.hadoop.fs.FileSystemLinkResolvert\000\033FileSystemLinkResolver.javat\000\aresolvesq\000~\000\023\000\000\001(t\000,org.apache.hadoop.hdfs.DistributedFileSystemq\000~\000Gq\000~\000Dsq\000~\000\023\000\000\002\374t\000\037org.apache.hadoop.fs.FileSystemt\000\017FileSystem.javaq\000~\000Dsq\000~\000\023\000\000\002ft\0008org.apache.reef.runtime.yarn.driver.YarnContainerManagert\000\031YarnContainerManager.javat\000\027appendByDeleteAndCreatesq\000~\000\023\000\000\002Wq\000~\000Tq\000~\000Ut\000\023writeToEvaluatorLogsq\000~\000\023\000\000\002{q\000~\000Tq\000~\000Ut\000\024logContainerAdditionsq\000~\000\023\000\000\001\317q\000~\000Tq\000~\000Ut\000\022handleNewContainersq\000~\000\023\000\000\000\204q\000~\000Tq\000~\000Ut\000\025onContainersAllocatedsq\000~\000\023\000\000\0014t\000Vorg.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$CallbackHandlerThreadt\000\030AMRMClientAsyncImpl.javat\000\003runsr\000&java.util.Collections$UnmodifiableList\374\017%1\265\354\216\020\002\000\001L\000\004listq\000~\000\005xr\000,java.util.Collections$UnmodifiableCollection\031B\000\200\313^\367\036\002\000\001L\000\001ct\000\026Ljava/util/Collection;xpsr\000\023java.util.ArrayListx\201\322\035\231\307a\235\003\000\001I\000\004sizexp\000\000\000\000w\004\000\000\000\000xq\000~\000hx\000\000\000\001t\000\035java.io.FileNotFoundExceptionq\000~\000\020uq\000~\000\021\000\000\000\030sq\000~\000\023\377\377\377\376t\000)sun.reflect.NativeConstructorAccessorImplt\000\"NativeConstructorAccessorImpl.javat\000\fnewInstance0sq\000~\000\023\000\000\0009q\000~\000lq\000~\000mt\000\vnewInstancesq\000~\000\023\000\000\000-t\000-sun.reflect.DelegatingConstructorAccessorImplt\000&DelegatingConstructorAccessorImpl.javaq\000~\000psq\000~\000\023\000\000\002\016t\000\035java.lang.reflect.Constructort\000\020Constructor.javaq\000~\000psq\000~\000\023\000\000\000jt\000%org.apache.hadoop.ipc.RemoteExceptiont\000\024RemoteException.javat\000\024instantiateExceptionsq\000~\000\023\000\000\000Iq\000~\000xq\000~\000yt\000\025unwrapRemoteExceptionsq\000~\000\023\000\000\004xq\000~\0005q\000~\0006q\000~\0007sq\000~\000\023\000\000\004lq\000~\0005q\000~\0006q\000~\0009sq\000~\000\023\000\000\004bq\000~\0005q\000~\0006q\000~\0009sq\000~\000\023\000\000\001\bq\000~\000<q\000~\000=q\000~\000>sq\000~\000\023\000\000\000\347q\000~\000<q\000~\000=q\000~\000@sq\000~\000\023\000\000\000\340q\000~\000<q\000~\000=q\000~\000Bsq\000~\000\023\000\000\005\017q\000~\0005q\000~\0006q\000~\000Dsq\000~\000\023\000\000\001,q\000~\000Fq\000~\000Gq\000~\000Hsq\000~\000\023\000\000\001(q\000~\000Fq\000~\000Gq\000~\000Hsq\000~\000\023\000\000\000Qq\000~\000Kq\000~\000Lq\000~\000Msq\000~\000\023\000\000\001(q\000~\000Oq\000~\000Gq\000~\000Dsq\000~\000\023\000\000\002\374q\000~\000Qq\000~\000Rq\000~\000Dsq\000~\000\023\000\000\002fq\000~\000Tq\000~\000Uq\000~\000Vsq\000~\000\023\000\000\002Wq\000~\000Tq\000~\000Uq\000~\000Xsq\000~\000\023\000\000\002{q\000~\000Tq\000~\000Uq\000~\000Zsq\000~\000\023\000\000\001\317q\000~\000Tq\000~\000Uq\000~\000\\sq\000~\000\023\000\000\000\204q\000~\000Tq\000~\000Uq\000~\000^sq\000~\000\023\000\000\0014q\000~\000`q\000~\000aq\000~\000bq\000~\000fxt\000:Cannot append by read-append-delete-create with exception.uq\000~\000\021\000\000\000\006sq\000~\000\023\000\000\002qq\000~\000Tq\000~\000Uq\000~\000Vsq\000~\000\023\000\000\002Wq\000~\000Tq\000~\000Uq\000~\000Xsq\000~\000\023\000\000\002{q\000~\000Tq\000~\000Uq\000~\000Zsq\000~\000\023\000\000\001\317q\000~\000Tq\000~\000Uq\000~\000\\sq\000~\000\023\000\000\000\204q\000~\000Tq\000~\000Uq\000~\000^sq\000~\000\023\000\000\0014q\000~\000`q\000~\000aq\000~\000bq\000~\000fxt\000Vjava.lang.RuntimeException: Cannot append by read-append-delete-create with exception.uq\000~\000\021\000\000\000\001sq\000~\000\023\000\000\001;q\000~\000`q\000~\000aq\000~\000bq\000~\000fxt\000@Thread AMRM Callback Handler Thread threw an uncaught exception.uq\000~\000\021\000\000\000\004sq\000~\000\023\000\000\0002t\000Borg.apache.reef.runtime.common.launch.REEFUncaughtExceptionHandlert\000!REEFUncaughtExceptionHandler.javat\000\021uncaughtExceptionsq\000~\000\023\000\000\004!t\000\025java.lang.ThreadGroupt\000\020ThreadGroup.javaq\000~\000\237sq\000~\000\023\000\000\004\034q\000~\000\241q\000~\000\242q\000~\000\237sq\000~\000\023\000\000\a\255t\000\020java.lang.Threadt\000\vThread.javat\000\031dispatchUncaughtExceptionq\000~\000fx"
identifier: "Vector"

2014-11-17 12:44:46,829 FINEST reef.wake.remote.impl.RemoteSenderEventHandler.onNext AMRM Callback Handler Thread | transport get link: localAddr: /147.46.215.66:41105 remoteAddr: /147.46.215.66:11066
2014-11-17 12:44:46,829 FINEST reef.wake.remote.impl.RemoteSenderEventHandler.setLink AMRM Callback Handler Thread | thread Thread[AMRM Callback Handler Thread,5,main] link localAddr: /147.46.215.66:41105 remoteAddr: /147.46.215.66:11066
java.lang.InterruptedException
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1219)
    at java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:340)
    at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:462)
    at org.apache.reef.wake.remote.impl.RemoteSenderEventHandler.consumeQueue(RemoteSenderEventHandler.java:74)
    at org.apache.reef.wake.remote.impl.RemoteSenderEventHandler.setLink(RemoteSenderEventHandler.java:68)
    at org.apache.reef.wake.remote.impl.RemoteSenderEventHandler.onNext(RemoteSenderEventHandler.java:99)
    at org.apache.reef.wake.remote.impl.RemoteSenderEventHandler.onNext(RemoteSenderEventHandler.java:40)
    at org.apache.reef.wake.remote.impl.ProxyEventHandler.onNext(ProxyEventHandler.java:73)
    at org.apache.reef.runtime.common.launch.REEFErrorHandler.onNext(REEFErrorHandler.java:76)
    at org.apache.reef.runtime.common.launch.REEFUncaughtExceptionHandler.uncaughtException(REEFUncaughtExceptionHandler.java:50)
    at java.lang.ThreadGroup.uncaughtException(ThreadGroup.java:1057)
    at java.lang.ThreadGroup.uncaughtException(ThreadGroup.java:1052)
    at java.lang.Thread.dispatchUncaughtException(Thread.java:1965)
2014-11-17 12:44:46,829 FINEST hadoop.ipc.ProtobufRpcEngine.invoke main | 1: Call -> cluster-07.ids.snu.ac.kr/147.46.215.84:8034: stopContainers {container_id { app_attempt_id { application_id { id: 29 cluster_timestamp: 1415768120886 } attemptId: 1 } id: 4 }}
org.apache.reef.wake.remote.exception.RemoteRuntimeException: java.lang.InterruptedException
    at org.apache.reef.wake.remote.impl.RemoteSenderEventHandler.consumeQueue(RemoteSenderEventHandler.java:80)
    at org.apache.reef.wake.remote.impl.RemoteSenderEventHandler.setLink(RemoteSenderEventHandler.java:68)
    at org.apache.reef.wake.remote.impl.RemoteSenderEventHandler.onNext(RemoteSenderEventHandler.java:99)
    at org.apache.reef.wake.remote.impl.RemoteSenderEventHandler.onNext(RemoteSenderEventHandler.java:40)
    at org.apache.reef.wake.remote.impl.ProxyEventHandler.onNext(ProxyEventHandler.java:73)
    at org.apache.reef.runtime.common.launch.REEFErrorHandler.onNext(REEFErrorHandler.java:76)
    at org.apache.reef.runtime.common.launch.REEFUncaughtExceptionHandler.uncaughtException(REEFUncaughtExceptionHandler.java:50)
    at java.lang.ThreadGroup.uncaughtException(ThreadGroup.java:1057)
    at java.lang.ThreadGroup.uncaughtException(ThreadGroup.java:1052)
    at java.lang.Thread.dispatchUncaughtException(Thread.java:1965)
Caused by: java.lang.InterruptedException
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1219)
    at java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:340)
    at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:462)
    at org.apache.reef.wake.remote.impl.RemoteSenderEventHandler.consumeQueue(RemoteSenderEventHandler.java:74)
    ... 9 more
2014-11-17 12:44:46,830 FINE hadoop.ipc.Client.run IPC Parameter Sending Thread #1 | IPC Client (1007777944) connection to cluster-07.ids.snu.ac.kr/147.46.215.84:8034 from appattempt_1415768120886_0029_000001 sending #64
2014-11-17 12:44:46,830 SEVERE reef.runtime.common.launch.REEFErrorHandler.onNext AMRM Callback Handler Thread | Unable to send the error upstream
org.apache.reef.wake.remote.exception.RemoteRuntimeException: java.lang.InterruptedException
    at org.apache.reef.wake.remote.impl.RemoteSenderEventHandler.consumeQueue(RemoteSenderEventHandler.java:80)
    at org.apache.reef.wake.remote.impl.RemoteSenderEventHandler.setLink(RemoteSenderEventHandler.java:68)
    at org.apache.reef.wake.remote.impl.RemoteSenderEventHandler.onNext(RemoteSenderEventHandler.java:99)
    at org.apache.reef.wake.remote.impl.RemoteSenderEventHandler.onNext(RemoteSenderEventHandler.java:40)
    at org.apache.reef.wake.remote.impl.ProxyEventHandler.onNext(ProxyEventHandler.java:73)
    at org.apache.reef.runtime.common.launch.REEFErrorHandler.onNext(REEFErrorHandler.java:76)
    at org.apache.reef.runtime.common.launch.REEFUncaughtExceptionHandler.uncaughtException(REEFUncaughtExceptionHandler.java:50)
    at java.lang.ThreadGroup.uncaughtException(ThreadGroup.java:1057)
    at java.lang.ThreadGroup.uncaughtException(ThreadGroup.java:1052)
    at java.lang.Thread.dispatchUncaughtException(Thread.java:1965)
Caused by: java.lang.InterruptedException
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1219)
    at java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:340)
    at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:462)
    at org.apache.reef.wake.remote.impl.RemoteSenderEventHandler.consumeQueue(RemoteSenderEventHandler.java:74)
    ... 9 more

2014-11-17 12:44:46,831 FINE hadoop.ipc.Client.receiveRpcResponse IPC Client (1007777944) connection to cluster-07.ids.snu.ac.kr/147.46.215.84:8034 from appattempt_1415768120886_0029_000001 | IPC Client (1007777944) connection to cluster-07.ids.snu.ac.kr/147.46.215.84:8034 from appattempt_1415768120886_0029_000001 got value #64
2014-11-17 12:44:46,831 FINE hadoop.ipc.ProtobufRpcEngine.invoke main | Call: stopContainers took 2ms
2014-11-17 12:44:46,835 FINEST hadoop.ipc.ProtobufRpcEngine.invoke main | 1: Response <- cluster-07.ids.snu.ac.kr/147.46.215.84:8034: stopContainers {succeeded_requests { app_attempt_id { application_id { id: 29 cluster_timestamp: 1415768120886 } attemptId: 1 } id: 4 }}
2014-11-17 12:44:46,835 FINEST hadoop.ipc.ProtobufRpcEngine.invoke main | 1: Call -> cluster-04.ids.snu.ac.kr/147.46.215.67:8034: stopContainers {container_id { app_attempt_id { application_id { id: 29 cluster_timestamp: 1415768120886 } attemptId: 1 } id: 3 }}
2014-11-17 12:44:46,835 FINE hadoop.ipc.Client.run IPC Parameter Sending Thread #1 | IPC Client (1007777944) connection to cluster-04.ids.snu.ac.kr/147.46.215.67:8034 from appattempt_1415768120886_0029_000001 sending #65
2014-11-17 12:44:46,837 FINE hadoop.ipc.Client.receiveRpcResponse IPC Client (1007777944) connection to cluster-04.ids.snu.ac.kr/147.46.215.67:8034 from appattempt_1415768120886_0029_000001 | IPC Client (1007777944) connection to cluster-04.ids.snu.ac.kr/147.46.215.67:8034 from appattempt_1415768120886_0029_000001 got value #65
2014-11-17 12:44:46,837 FINE hadoop.ipc.ProtobufRpcEngine.invoke main | Call: stopContainers took 2ms
2014-11-17 12:44:46,840 FINEST hadoop.ipc.ProtobufRpcEngine.invoke main | 1: Response <- cluster-04.ids.snu.ac.kr/147.46.215.67:8034: stopContainers {succeeded_requests { app_attempt_id { application_id { id: 29 cluster_timestamp: 1415768120886 } attemptId: 1 } id: 3 }}
2014-11-17 12:44:46,840 FINEST hadoop.ipc.ProtobufRpcEngine.invoke main | 1: Call -> cluster-07.ids.snu.ac.kr/147.46.215.84:8034: stopContainers {container_id { app_attempt_id { application_id { id: 29 cluster_timestamp: 1415768120886 } attemptId: 1 } id: 2 }}
2014-11-17 12:44:46,840 FINE hadoop.ipc.Client.run IPC Parameter Sending Thread #1 | IPC Client (1007777944) connection to cluster-07.ids.snu.ac.kr/147.46.215.84:8034 from appattempt_1415768120886_0029_000001 sending #66
2014-11-17 12:44:46,842 FINE hadoop.ipc.Client.receiveRpcResponse IPC Client (1007777944) connection to cluster-07.ids.snu.ac.kr/147.46.215.84:8034 from appattempt_1415768120886_0029_000001 | IPC Client (1007777944) connection to cluster-07.ids.snu.ac.kr/147.46.215.84:8034 from appattempt_1415768120886_0029_000001 got value #66
2014-11-17 12:44:46,842 FINE hadoop.ipc.ProtobufRpcEngine.invoke main | Call: stopContainers took 2ms
2014-11-17 12:44:46,842 FINEST hadoop.ipc.ProtobufRpcEngine.invoke main | 1: Response <- cluster-07.ids.snu.ac.kr/147.46.215.84:8034: stopContainers {succeeded_requests { app_attempt_id { application_id { id: 29 cluster_timestamp: 1415768120886 } attemptId: 1 } id: 2 }}
2014-11-17 12:44:46,842 INFO hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy.mayBeCloseProxy main | Closing proxy : cluster-04.ids.snu.ac.kr:8034
2014-11-17 12:44:46,843 FINE hadoop.ipc.Client.stopClient main | stopping client from cache: org.apache.hadoop.ipc.Client@3fde1ed4
2014-11-17 12:44:46,843 INFO hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy.mayBeCloseProxy main | Closing proxy : cluster-07.ids.snu.ac.kr:8034
2014-11-17 12:44:46,843 FINE hadoop.ipc.Client.stopClient main | stopping client from cache: org.apache.hadoop.ipc.Client@3fde1ed4
2014-11-17 12:44:46,843 FINEST reef.wake.impl.PubSubEventHandler.onNext main | Invoking org.apache.reef.runtime.common.driver.DriverRuntimeStopHandler@52d5353
2014-11-17 12:44:46,843 FINEST reef.runtime.common.driver.DriverRuntimeStopHandler.onNext main | RuntimeStop: org.apache.reef.wake.time.runtime.event.RuntimeStop[1416195886823]
2014-11-17 12:44:46,844 WARNING reef.runtime.common.driver.evaluator.Evaluators.close main | Unclean shutdown of evaluator container_1415768120886_0029_01_000004
2014-11-17 12:44:46,844 WARNING reef.runtime.common.driver.evaluator.EvaluatorManager.close main | Force resource release because the client closed the clock.
java.lang.IllegalStateException: Scheduling alarm on a closed clock
    at org.apache.reef.wake.time.runtime.RuntimeClock.scheduleAlarm(RuntimeClock.java:80)
    at org.apache.reef.runtime.common.driver.evaluator.EvaluatorManager.close(EvaluatorManager.java:201)
    at org.apache.reef.runtime.common.driver.evaluator.Evaluators.close(Evaluators.java:69)
    at org.apache.reef.runtime.common.driver.DriverRuntimeStopHandler.onNext(DriverRuntimeStopHandler.java:59)
    at org.apache.reef.runtime.common.driver.DriverRuntimeStopHandler.onNext(DriverRuntimeStopHandler.java:37)
    at org.apache.reef.wake.impl.PubSubEventHandler.onNext(PubSubEventHandler.java:98)
    at org.apache.reef.wake.time.runtime.RuntimeClock.run(RuntimeClock.java:231)
    at org.apache.reef.runtime.common.launch.LaunchClass.run(LaunchClass.java:162)
    at org.apache.reef.runtime.common.Launcher.main(Launcher.java:111)

2014-11-17 12:44:46,844 FINEST reef.runtime.yarn.driver.YARNResourceReleaseHandler.onNext main | Releasing container container_1415768120886_0029_01_000004
2014-11-17 12:44:46,844 FINE reef.runtime.yarn.driver.YarnContainerManager.release main | Release container: container_1415768120886_0029_01_000004
2014-11-17 12:44:46,855 FINER reef.wake.remote.transport.netty.AbstractNettyEventListener.closeChannel org.apache.reef.wake.remote.transport.netty.NettyMessagingTransportServerWorker-pool-4-thread-1 | Channel closed: [id: 0x0062439d, /147.46.215.84:59736 :> /147.46.215.66:14140]. Link ref found and removed: true
2014-11-17 12:44:46,862 FINEST hadoop.ipc.ProtobufRpcEngine.invoke main | 1: Call -> cluster-01.ids.snu.ac.kr/147.46.215.66:9000: getFileInfo {src: "/ReefApplications/application_1415768120886_0029/evaluatorsChangesLog"}
2014-11-17 12:44:46,863 FINE hadoop.ipc.Client.run IPC Parameter Sending Thread #1 | IPC Client (1007777944) connection to cluster-01.ids.snu.ac.kr/147.46.215.66:9000 from hadoop sending #67
2014-11-17 12:44:46,863 FINE hadoop.ipc.Client.receiveRpcResponse IPC Client (1007777944) connection to cluster-01.ids.snu.ac.kr/147.46.215.66:9000 from hadoop | IPC Client (1007777944) connection to cluster-01.ids.snu.ac.kr/147.46.215.66:9000 from hadoop got value #67
2014-11-17 12:44:46,864 FINE hadoop.ipc.ProtobufRpcEngine.invoke main | Call: getFileInfo took 2ms
2014-11-17 12:44:46,864 FINEST hadoop.ipc.ProtobufRpcEngine.invoke main | 1: Response <- cluster-01.ids.snu.ac.kr/147.46.215.66:9000: getFileInfo {fs { fileType: IS_FILE path: "" length: 120 permission { perm: 420 } owner: "hadoop" group: "supergroup" modification_time: 1416195886741 access_time: 1416195886607 block_replication: 3 blocksize: 134217728 fileId: 19939 childrenNum: 0 }}
2014-11-17 12:44:46,865 FINEST hadoop.ipc.ProtobufRpcEngine.invoke main | 1: Call -> cluster-01.ids.snu.ac.kr/147.46.215.66:9000: getFileInfo {src: "/ReefApplications/application_1415768120886_0029/evaluatorsChangesLog"}
2014-11-17 12:44:46,865 FINE hadoop.ipc.Client.run IPC Parameter Sending Thread #1 | IPC Client (1007777944) connection to cluster-01.ids.snu.ac.kr/147.46.215.66:9000 from hadoop sending #68
2014-11-17 12:44:46,866 FINE hadoop.ipc.Client.receiveRpcResponse IPC Client (1007777944) connection to cluster-01.ids.snu.ac.kr/147.46.215.66:9000 from hadoop | IPC Client (1007777944) connection to cluster-01.ids.snu.ac.kr/147.46.215.66:9000 from hadoop got value #68
2014-11-17 12:44:46,866 FINE hadoop.ipc.ProtobufRpcEngine.invoke main | Call: getFileInfo took 1ms
2014-11-17 12:44:46,866 FINEST hadoop.ipc.ProtobufRpcEngine.invoke main | 1: Response <- cluster-01.ids.snu.ac.kr/147.46.215.66:9000: getFileInfo {fs { fileType: IS_FILE path: "" length: 120 permission { perm: 420 } owner: "hadoop" group: "supergroup" modification_time: 1416195886741 access_time: 1416195886607 block_replication: 3 blocksize: 134217728 fileId: 19939 childrenNum: 0 }}
2014-11-17 12:44:46,867 FINEST hadoop.ipc.ProtobufRpcEngine.invoke main | 1: Call -> cluster-01.ids.snu.ac.kr/147.46.215.66:9000: append {src: "/ReefApplications/application_1415768120886_0029/evaluatorsChangesLog" clientName: "DFSClient_NONMAPREDUCE_1658614091_1"}
2014-11-17 12:44:46,867 FINE hadoop.ipc.Client.run IPC Parameter Sending Thread #1 | IPC Client (1007777944) connection to cluster-01.ids.snu.ac.kr/147.46.215.66:9000 from hadoop sending #69
2014-11-17 12:44:46,882 FINE hadoop.ipc.Client.receiveRpcResponse IPC Client (1007777944) connection to cluster-01.ids.snu.ac.kr/147.46.215.66:9000 from hadoop | IPC Client (1007777944) connection to cluster-01.ids.snu.ac.kr/147.46.215.66:9000 from hadoop got value #69
2014-11-17 12:44:46,882 FINE hadoop.ipc.ProtobufRpcEngine.invoke main | Call: append took 15ms
2014-11-17 12:44:46,884 FINEST hadoop.ipc.ProtobufRpcEngine.invoke main | 1: Response <- cluster-01.ids.snu.ac.kr/147.46.215.66:9000: append {block { b { poolId: "BP-359027977-147.46.215.66-1414669643463" blockId: 1073745062 generationStamp: 9952 numBytes: 120 } offset: 0 locs { id { ipAddr: "147.46.215.66" hostName: "cluster-01.ids.snu.ac.kr" datanodeUuid: "cfa5e522-188d-4388-919b-5d84f559bfc8" xferPort: 50010 infoPort: 50075 ipcPort: 50020 infoSecurePort: 0 } capacity: 1957028253696 dfsUsed: 3170075493 remaining: 791286038528 blockPoolUsed: 3170075493 lastUpdate: 1416195884982 xceiverCount: 2 location: "/default-rack" adminState: NORMAL cacheCapacity: 0 cacheUsed: 0 } locs { id { ipAddr: "147.46.215.75" hostName: "cluster-06.ids.snu.ac.kr" datanodeUuid: "c16fb521-163a-4b2c-bf4b-8b45a5cd0196" xferPort: 50010 infoPort: 50075 ipcPort: 50020 infoSecurePort: 0 } capacity: 972527087616 dfsUsed: 833897728 remaining: 691221901312 blockPoolUsed: 833897728 lastUpdate: 1416195884765 xceiverCount: 1 location: "/default-rack" adminState: NORMAL cacheCapacity: 0 cacheUsed: 0 } locs { id { ipAddr: "147.46.215.68" hostName: "cluster-03.ids.snu.ac.kr" datanodeUuid: "06043d3e-777b-4b76-901a-d43ed7f4ec37" xferPort: 50010 infoPort: 50075 ipcPort: 50020 infoSecurePort: 0 } capacity: 1957028253696 dfsUsed: 955294390 remaining: 1648132288512 blockPoolUsed: 955294390 lastUpdate: 1416195884773 xceiverCount: 1 location: "/default-rack" adminState: NORMAL cacheCapacity: 0 cacheUsed: 0 } corrupt: false blockToken { identifier: "" password: "" kind: "" service: "" } isCached: false isCached: false isCached: false storageTypes: DISK storageTypes: DISK storageTypes: DISK storageIDs: "DS-2108747149-147.46.215.66-50010-1380010322582" storageIDs: "DS-429422048-147.46.215.75-50010-1380010329366" storageIDs: "DS-825774162-147.46.215.68-50010-1380010347494" }}
2014-11-17 12:44:46,884 FINE hadoop.hdfs.DFSClient.computePacketChunkSize main | computePacketChunkSize: src=/ReefApplications/application_1415768120886_0029/evaluatorsChangesLog, chunkSize=396, chunksPerPacket=1, packetSize=396
2014-11-17 12:44:46,885 FINE hadoop.hdfs.DFSClient.writeChunk main | DFSClient writeChunk allocating new packet seqno=0, src=/ReefApplications/application_1415768120886_0029/evaluatorsChangesLog, packetSize=396, chunksPerPacket=1, bytesCurBlock=120
2014-11-17 12:44:46,885 FINE hadoop.hdfs.DFSClient.writeChunk main | DFSClient writeChunk packet full seqno=0, src=/ReefApplications/application_1415768120886_0029/evaluatorsChangesLog, bytesCurBlock=160, blockSize=134217728, appendChunk=true
2014-11-17 12:44:46,885 FINE hadoop.hdfs.DFSClient.queueCurrentPacket main | Queued packet 0
2014-11-17 12:44:46,885 FINE hadoop.hdfs.DFSClient.queueCurrentPacket main | Queued packet 1
2014-11-17 12:44:46,885 FINE hadoop.hdfs.DFSClient.run Thread-32 | Append to block BP-359027977-147.46.215.66-1414669643463:blk_1073745062_9952
2014-11-17 12:44:46,885 FINE hadoop.hdfs.DFSClient.waitForAckedSeqno main | Waiting for ack for: 1
2014-11-17 12:44:46,886 FINEST hadoop.ipc.ProtobufRpcEngine.invoke Thread-32 | 191: Call -> cluster-01.ids.snu.ac.kr/147.46.215.66:9000: updateBlockForPipeline {block { poolId: "BP-359027977-147.46.215.66-1414669643463" blockId: 1073745062 generationStamp: 9952 numBytes: 120 } clientName: "DFSClient_NONMAPREDUCE_1658614091_1"}
2014-11-17 12:44:46,886 FINE hadoop.ipc.Client.run IPC Parameter Sending Thread #1 | IPC Client (1007777944) connection to cluster-01.ids.snu.ac.kr/147.46.215.66:9000 from hadoop sending #70
2014-11-17 12:44:46,890 FINE hadoop.ipc.Client.receiveRpcResponse IPC Client (1007777944) connection to cluster-01.ids.snu.ac.kr/147.46.215.66:9000 from hadoop | IPC Client (1007777944) connection to cluster-01.ids.snu.ac.kr/147.46.215.66:9000 from hadoop got value #70
2014-11-17 12:44:46,891 FINE hadoop.ipc.ProtobufRpcEngine.invoke Thread-32 | Call: updateBlockForPipeline took 6ms
2014-11-17 12:44:46,893 FINEST hadoop.ipc.ProtobufRpcEngine.invoke Thread-32 | 191: Response <- cluster-01.ids.snu.ac.kr/147.46.215.66:9000: updateBlockForPipeline {block { b { poolId: "BP-359027977-147.46.215.66-1414669643463" blockId: 1073745062 generationStamp: 9953 numBytes: 120 } offset: 18446744073709551615 corrupt: false blockToken { identifier: "" password: "" kind: "" service: "" } }}
2014-11-17 12:44:46,893 FINE hadoop.hdfs.DFSClient.createBlockOutputStream Thread-32 | pipeline = 147.46.215.66:50010
2014-11-17 12:44:46,893 FINE hadoop.hdfs.DFSClient.createBlockOutputStream Thread-32 | pipeline = 147.46.215.75:50010
2014-11-17 12:44:46,894 FINE hadoop.hdfs.DFSClient.createBlockOutputStream Thread-32 | pipeline = 147.46.215.68:50010
2014-11-17 12:44:46,894 FINE hadoop.hdfs.DFSClient.createSocketForPipeline Thread-32 | Connecting to datanode 147.46.215.66:50010
2014-11-17 12:44:46,894 FINE hadoop.hdfs.DFSClient.createSocketForPipeline Thread-32 | Send buf size 131071
2014-11-17 12:44:46,895 FINEST hadoop.hdfs.protocol.datatransfer.DataTransferProtocol.send Thread-32 | Sending DataTransferOp OpWriteBlockProto: header {
  baseHeader {
    block {
      poolId: "BP-359027977-147.46.215.66-1414669643463"
      blockId: 1073745062
      generationStamp: 9952
      numBytes: 120
    }
    token {
      identifier: ""
      password: ""
      kind: ""
      service: ""
    }
  }
  clientName: "DFSClient_NONMAPREDUCE_1658614091_1"
}
targets {
  id {
    ipAddr: "147.46.215.75"
    hostName: "cluster-06.ids.snu.ac.kr"
    datanodeUuid: "c16fb521-163a-4b2c-bf4b-8b45a5cd0196"
    xferPort: 50010
    infoPort: 50075
    ipcPort: 50020
    infoSecurePort: 0
  }
  capacity: 972527087616
  dfsUsed: 833897728
  remaining: 691221901312
  blockPoolUsed: 833897728
  lastUpdate: 1416195884765
  xceiverCount: 1
  location: "/default-rack"
  adminState: NORMAL
  cacheCapacity: 0
  cacheUsed: 0
}
targets {
  id {
    ipAddr: "147.46.215.68"
    hostName: "cluster-03.ids.snu.ac.kr"
    datanodeUuid: "06043d3e-777b-4b76-901a-d43ed7f4ec37"
    xferPort: 50010
    infoPort: 50075
    ipcPort: 50020
    infoSecurePort: 0
  }
  capacity: 1957028253696
  dfsUsed: 955294390
  remaining: 1648132288512
  blockPoolUsed: 955294390
  lastUpdate: 1416195884773
  xceiverCount: 1
  location: "/default-rack"
  adminState: NORMAL
  cacheCapacity: 0
  cacheUsed: 0
}
stage: PIPELINE_SETUP_APPEND
pipelineSize: 3
minBytesRcvd: 120
maxBytesRcvd: 120
latestGenerationStamp: 9953
requestedChecksum {
  type: CHECKSUM_CRC32C
  bytesPerChecksum: 512
}
cachingStrategy {
}

2014-11-17 12:44:46,908 FINEST hadoop.ipc.ProtobufRpcEngine.invoke Thread-32 | 191: Call -> cluster-01.ids.snu.ac.kr/147.46.215.66:9000: updatePipeline {clientName: "DFSClient_NONMAPREDUCE_1658614091_1" oldBlock { poolId: "BP-359027977-147.46.215.66-1414669643463" blockId: 1073745062 generationStamp: 9952 numBytes: 120 } newBlock { poolId: "BP-359027977-147.46.215.66-1414669643463" blockId: 1073745062 generationStamp: 9953 numBytes: 120 } newNodes { ipAddr: "147.46.215.66" hostName: "cluster-01.ids.snu.ac.kr" datanodeUuid: "cfa5e522-188d-4388-919b-5d84f559bfc8" xferPort: 50010 infoPort: 50075 ipcPort: 50020 infoSecurePort: 0 } newNodes { ipAddr: "147.46.215.75" hostName: "cluster-06.ids.snu.ac.kr" datanodeUuid: "c16fb521-163a-4b2c-bf4b-8b45a5cd0196" xferPort: 50010 infoPort: 50075 ipcPort: 50020 infoSecurePort: 0 } newNodes { ipAddr: "147.46.215.68" hostName: "cluster-03.ids.snu.ac.kr" datanodeUuid: "06043d3e-777b-4b76-901a-d43ed7f4ec37" xferPort: 50010 infoPort: 50075 ipcPort: 50020 infoSecurePort: 0 } storageIDs: "DS-2108747149-147.46.215.66-50010-1380010322582" storageIDs: "DS-429422048-147.46.215.75-50010-1380010329366" storageIDs: "DS-825774162-147.46.215.68-50010-1380010347494"}
2014-11-17 12:44:46,908 FINE hadoop.ipc.Client.run IPC Parameter Sending Thread #1 | IPC Client (1007777944) connection to cluster-01.ids.snu.ac.kr/147.46.215.66:9000 from hadoop sending #71
2014-11-17 12:44:46,915 FINE hadoop.ipc.Client.receiveRpcResponse IPC Client (1007777944) connection to cluster-01.ids.snu.ac.kr/147.46.215.66:9000 from hadoop | IPC Client (1007777944) connection to cluster-01.ids.snu.ac.kr/147.46.215.66:9000 from hadoop got value #71
2014-11-17 12:44:46,915 FINE hadoop.ipc.ProtobufRpcEngine.invoke Thread-32 | Call: updatePipeline took 8ms
2014-11-17 12:44:46,916 FINEST hadoop.ipc.ProtobufRpcEngine.invoke Thread-32 | 191: Response <- cluster-01.ids.snu.ac.kr/147.46.215.66:9000: updatePipeline {}
2014-11-17 12:44:46,916 FINE hadoop.hdfs.DFSClient.run DataStreamer for file /ReefApplications/application_1415768120886_0029/evaluatorsChangesLog block BP-359027977-147.46.215.66-1414669643463:blk_1073745062_9953 | DataStreamer block BP-359027977-147.46.215.66-1414669643463:blk_1073745062_9953 sending packet packet seqno:0 offsetInBlock:120 lastPacketInBlock:false lastByteOffsetInBlock: 160
2014-11-17 12:44:46,918 FINE hadoop.hdfs.DFSClient.run ResponseProcessor for block BP-359027977-147.46.215.66-1414669643463:blk_1073745062_9953 | DFSClient seqno: 0 status: SUCCESS status: SUCCESS status: SUCCESS downstreamAckTimeNanos: 1168462
2014-11-17 12:44:46,918 FINE hadoop.hdfs.DFSClient.run DataStreamer for file /ReefApplications/application_1415768120886_0029/evaluatorsChangesLog block BP-359027977-147.46.215.66-1414669643463:blk_1073745062_9953 | DataStreamer block BP-359027977-147.46.215.66-1414669643463:blk_1073745062_9953 sending packet packet seqno:1 offsetInBlock:160 lastPacketInBlock:true lastByteOffsetInBlock: 160
2014-11-17 12:44:46,920 FINE hadoop.hdfs.DFSClient.run ResponseProcessor for block BP-359027977-147.46.215.66-1414669643463:blk_1073745062_9953 | DFSClient seqno: 1 status: SUCCESS status: SUCCESS status: SUCCESS downstreamAckTimeNanos: 1027373
2014-11-17 12:44:46,922 FINE hadoop.hdfs.DFSClient.endBlock DataStreamer for file /ReefApplications/application_1415768120886_0029/evaluatorsChangesLog block BP-359027977-147.46.215.66-1414669643463:blk_1073745062_9953 | Closing old block BP-359027977-147.46.215.66-1414669643463:blk_1073745062_9953
2014-11-17 12:44:46,923 FINEST hadoop.ipc.ProtobufRpcEngine.invoke main | 1: Call -> cluster-01.ids.snu.ac.kr/147.46.215.66:9000: complete {src: "/ReefApplications/application_1415768120886_0029/evaluatorsChangesLog" clientName: "DFSClient_NONMAPREDUCE_1658614091_1" last { poolId: "BP-359027977-147.46.215.66-1414669643463" blockId: 1073745062 generationStamp: 9953 numBytes: 160 } fileId: 19939}
2014-11-17 12:44:46,923 FINE hadoop.ipc.Client.run IPC Parameter Sending Thread #1 | IPC Client (1007777944) connection to cluster-01.ids.snu.ac.kr/147.46.215.66:9000 from hadoop sending #72
2014-11-17 12:44:46,930 FINE reef.wake.remote.impl.DefaultRemoteManagerImplementation.close AMRM Callback Handler Thread | RemoteManager: REEF_LAUNCHER Closing remote manager id: socket://147.46.215.66:14140
2014-11-17 12:44:46,932 FINE reef.wake.remote.impl.DefaultRemoteManagerImplementation$1.run pool-5-thread-1 | Closing sender stage socket://147.46.215.66:14140
2014-11-17 12:44:46,932 FINE reef.wake.remote.impl.RemoteSenderStage.close pool-5-thread-1 | close org.apache.reef.wake.remote.transport.netty.NettyMessagingTransport@7ef0776c
2014-11-17 12:44:46,932 FINE hadoop.ipc.Client.receiveRpcResponse IPC Client (1007777944) connection to cluster-01.ids.snu.ac.kr/147.46.215.66:9000 from hadoop | IPC Client (1007777944) connection to cluster-01.ids.snu.ac.kr/147.46.215.66:9000 from hadoop got value #72
2014-11-17 12:44:46,932 FINE reef.wake.remote.impl.DefaultRemoteManagerImplementation$1.run pool-5-thread-1 | Closed the remote sender stage
2014-11-17 12:44:46,932 FINE reef.wake.remote.impl.DefaultRemoteManagerImplementation$1.run pool-5-thread-1 | Closing transport socket://147.46.215.66:14140
2014-11-17 12:44:46,933 FINE hadoop.ipc.ProtobufRpcEngine.invoke main | Call: complete took 10ms
2014-11-17 12:44:46,933 FINE reef.wake.remote.transport.netty.NettyMessagingTransport.close pool-5-thread-1 | Closing netty transport socket address: /147.46.215.66:14140
2014-11-17 12:44:46,933 FINEST hadoop.ipc.ProtobufRpcEngine.invoke main | 1: Response <- cluster-01.ids.snu.ac.kr/147.46.215.66:9000: complete {result: true}
2014-11-17 12:44:46,933 FINEST reef.runtime.common.driver.resourcemanager.ResourceManagerStatus.onNext main | Runtime status name: "YARN"
state: RUNNING
outstanding_container_requests: 3
container_allocation: "container_1415768120886_0029_01_000003"

2014-11-17 12:44:46,934 WARNING reef.runtime.common.driver.evaluator.Evaluators.close main | Unclean shutdown of evaluator container_1415768120886_0029_01_000002
2014-11-17 12:44:46,934 WARNING reef.runtime.common.driver.evaluator.Evaluators.close main | Unclean shutdown of evaluator container_1415768120886_0029_01_000003
2014-11-17 12:44:46,934 WARNING reef.runtime.common.driver.evaluator.EvaluatorManager.close main | Force resource release because the client closed the clock.
java.lang.IllegalStateException: Scheduling alarm on a closed clock
    at org.apache.reef.wake.time.runtime.RuntimeClock.scheduleAlarm(RuntimeClock.java:80)
    at org.apache.reef.runtime.common.driver.evaluator.EvaluatorManager.close(EvaluatorManager.java:201)
    at org.apache.reef.runtime.common.driver.evaluator.Evaluators.close(Evaluators.java:69)
    at org.apache.reef.runtime.common.driver.DriverRuntimeStopHandler.onNext(DriverRuntimeStopHandler.java:59)
    at org.apache.reef.runtime.common.driver.DriverRuntimeStopHandler.onNext(DriverRuntimeStopHandler.java:37)
    at org.apache.reef.wake.impl.PubSubEventHandler.onNext(PubSubEventHandler.java:98)
    at org.apache.reef.wake.time.runtime.RuntimeClock.run(RuntimeClock.java:231)
    at org.apache.reef.runtime.common.launch.LaunchClass.run(LaunchClass.java:162)
    at org.apache.reef.runtime.common.Launcher.main(Launcher.java:111)

2014-11-17 12:44:46,934 FINEST reef.runtime.yarn.driver.YARNResourceReleaseHandler.onNext main | Releasing container container_1415768120886_0029_01_000003
2014-11-17 12:44:46,935 FINE reef.runtime.yarn.driver.YarnContainerManager.release main | Release container: container_1415768120886_0029_01_000003
2014-11-17 12:44:46,940 FINER reef.wake.remote.transport.netty.AbstractNettyEventListener.closeChannel org.apache.reef.wake.remote.transport.netty.NettyMessagingTransportClientWorker-pool-5-thread-1 | Channel closed: [id: 0x758d5201, /147.46.215.66:41105 :> /147.46.215.66:11066]. Link ref found and removed: true
2014-11-17 12:44:46,946 FINE reef.wake.remote.transport.netty.NettyMessagingTransport.close pool-5-thread-1 | Closing netty transport socket address: /147.46.215.66:14140 done
2014-11-17 12:44:46,946 FINE reef.wake.remote.impl.DefaultRemoteManagerImplementation$1.run pool-5-thread-1 | Closed the transport
2014-11-17 12:44:46,946 FINE reef.wake.remote.impl.DefaultRemoteManagerImplementation$1.run pool-5-thread-1 | Closing receiver stage socket://147.46.215.66:14140
2014-11-17 12:44:46,946 FINE reef.wake.remote.impl.OrderedRemoteReceiverStage.close pool-5-thread-1 | close
2014-11-17 12:44:46,948 FINE reef.wake.remote.impl.DefaultRemoteManagerImplementation$1.run pool-5-thread-1 | Closed the remote receiver stage
2014-11-17 12:44:46,948 FINE reef.wake.remote.impl.DefaultRemoteManagerImplementation.close AMRM Callback Handler Thread | Close executor terminated properly.
2014-11-17 12:44:46,948 SEVERE reef.runtime.common.launch.REEFUncaughtExceptionHandler.uncaughtException AMRM Callback Handler Thread | System.exit(1)
2014-11-17 12:44:46,949 FINEST reef.wake.impl.StageManager$1.run Thread-1 | Shutdown hook : closing stages
bchocho commented 9 years ago

@kim333 @hanmanhui Thanks. This part looks interesting:

Caused by: java.io.FileNotFoundException: File does not exist: /ReefApplications/application_1415768120886_0029/evaluatorsChangesLog

Could this be a write permissions issue on HDFS? Can you check with hdfs dfs -ls:

Also, what user are you running the Hadoop jobs as?

kim333 commented 9 years ago
  1. /ReefApplications/ exists
  2. /ReefApplications/application_1415768120886_0029 exists
  3. /ReefApplications/application_1415768120886_0029/evaluatorsChangesLog exists
  4. Owner of the directories is hadoop user and the group is supergroup and the permission is drwxr-xr-x
  5. We are executing the jobs on user hadoop.