search

apache / linkis

Apache Linkis builds a computation middleware layer to facilitate connection, governance and orchestration between the upper applications and the underlying data engines.
https://linkis.apache.org/
Apache License 2.0
3.3k stars 1.17k forks source link

Failed to APPEND FILE because of hdfs lease #333

Closed whyreal closed 4 years ago

whyreal commented 4 years ago

logs from bml-server when i saving a workflow.

2020-03-24 17:23:50.023 ERROR [qtp1501976525-22] com.webank.wedatasphere.linkis.server.restful.RestfulCatchAOP 72 apply - error code(错误码): 60050, error message(错误信息): 首次上传资源失败. com.webank.wedatasphere.linkis.bml.common.UploadResourceException: errCode: 60050 ,desc: 首次上传资源失败 ,ip: hadoop5.sxzkzy.local ,port: 9113 ,serviceKind: bml-server
        at com.webank.wedatasphere.linkis.bml.common.HdfsResourceHelper.upload(HdfsResourceHelper.java:75) ~[linkis-bmlserver-0.9.3.jar:?]
        at com.webank.wedatasphere.linkis.bml.service.impl.VersionServiceImpl.updateVersion(VersionServiceImpl.java:106) ~[linkis-bmlserver-0.9.3.jar:?]
        at com.webank.wedatasphere.linkis.bml.service.impl.TaskServiceImpl.createUpdateTask(TaskServiceImpl.java:123) ~[linkis-bmlserver-0.9.3.jar:?]
        at com.webank.wedatasphere.linkis.bml.restful.BmlRestfulApi.updateVersion(BmlRestfulApi.java:503) ~[linkis-bmlserver-0.9.3.jar:?]
        at com.webank.wedatasphere.linkis.bml.restful.BmlRestfulApi$$FastClassBySpringCGLIB$$371fe1e8.invoke(<generated>) ~[linkis-bmlserver-0.9.3.jar:?]
        at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204) ~[spring-core-5.0.7.RELEASE.jar:5.0.7.RELEASE]
        at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:746) ~[spring-aop-5.0.7.RELEASE.jar:5.0.7.RELEASE]
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) ~[spring-aop-5.0.7.RELEASE.jar:5.0.7.RELEASE]
        at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:88) ~[spring-aop-5.0.7.RELEASE.jar:5.0.7.RELEASE]
        at com.webank.wedatasphere.linkis.server.restful.RestfulCatchAOP$$anonfun$1.apply(RestfulCatchAOP.scala:48) ~[linkis-module-0.9.3.jar:?]
        at com.webank.wedatasphere.linkis.server.restful.RestfulCatchAOP$$anonfun$1.apply(RestfulCatchAOP.scala:48) ~[linkis-module-0.9.3.jar:?]
        at com.webank.wedatasphere.linkis.common.utils.Utils$.tryCatch(Utils.scala:48) [linkis-common-0.9.3.jar:?]
        at com.webank.wedatasphere.linkis.server.package$.catchMsg(package.scala:57) [linkis-module-0.9.3.jar:?]
        at com.webank.wedatasphere.linkis.server.package$.catchIt(package.scala:89) [linkis-module-0.9.3.jar:?]
        at com.webank.wedatasphere.linkis.server.restful.RestfulCatchAOP.dealResponseRestful(RestfulCatchAOP.scala:47) [linkis-module-0.9.3.jar:?]
........
Caused by: org.apache.hadoop.ipc.RemoteException: Failed to APPEND_FILE /tmp/linkis/hadoop/20200324/158504147872250.sh for DFSClient_NONMAPREDUCE_27966040_22 on 192.168.80.30 because this file lease is currently owned by DFSClient_NONMAPREDUCE_-1168761573_25 on 192.168.80.30
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLeaseInternal(FSNamesystem.java:2604)
        at org.apache.hadoop.hdfs.server.namenode.FSDirAppendOp.appendFile(FSDirAppendOp.java:124)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.appendFile(FSNamesystem.java:2639)
        at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.append(NameNodeRpcServer.java:805)
        at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.append(ClientNamenodeProtocolServerSideTranslatorPB.java:487)
        at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
        ... 107 more
peacewong commented 4 years ago

Hello, this is the problem of hdfs