Apache Linkis builds a computation middleware layer to facilitate connection, governance and orchestration between the upper applications and the underlying data engines.
when run com.webank.wedatasphere.linkis.ujes.client.LinkisClientTest,
exception throw when run code String resultSet = jobInfo.getResultSetList(client)[0],
the linkis-cg-entrance throw a exception about filesystem;
error debug:
entrance error log
2021-03-30 16:48:30.002 [WARN ] [linkisClient-Test_geosmart_ops_sparkConsumerThread] c.w.w.l.e.l.CacheLogManager (55) [apply] -
write log for job linkisClient-Test_geosmart_ops_spark_6 failed java.io.IOException: Failed on local exception:
java.nio.channels.ClosedByInterruptException; Host Details :
local host is: "host1";
destination host is: "host2":8020;
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:772) ~[hadoop-common-2.6.5.jar:?]
at org.apache.hadoop.ipc.Client.call(Client.java:1474) ~[hadoop-common-2.6.5.jar:?]
at org.apache.hadoop.ipc.Client.call(Client.java:1401) ~[hadoop-common-2.6.5.jar:?]
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232) ~[hadoop-common-2.6.5.jar:?]
at com.sun.proxy.$Proxy232.getFileInfo(Unknown Source) ~[?:?]
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:752) ~[hadoop-hdfs-2.6.5.jar:?]
at sun.reflect.GeneratedMethodAccessor66.invoke(Unknown Source) ~[?:?]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_275]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_275]
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187) ~[hadoop-common-2.6.5.jar:?]
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) ~[hadoop-common-2.6.5.jar:?]
at com.sun.proxy.$Proxy233.getFileInfo(Unknown Source) ~[?:?]
at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1977) ~[hadoop-hdfs-2.6.5.jar:?]
at org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1118) ~[hadoop-hdfs-2.6.5.jar:?]
at org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1114) ~[hadoop-hdfs-2.6.5.jar:?]
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) ~[hadoop-common-2.6.5.jar:?]
at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1114) ~[hadoop-hdfs-2.6.5.jar:?]
at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1400) ~[hadoop-common-2.6.5.jar:?]
at com.webank.wedatasphere.linkis.storage.fs.impl.HDFSFileSystem.exists(HDFSFileSystem.java:278) ~[linkis-storage-1.0.0-RC1.jar:?]
at com.webank.wedatasphere.linkis.storage.utils.FileSystemUtils$$anonfun$createNewFile$4.apply(FileSystemUtils.scala:68) ~[linkis-storage-1.0.0-RC1.jar:?]
at com.webank.wedatasphere.linkis.common.utils.Utils$.tryFinally(Utils.scala:62) ~[linkis-common-1.0.0-RC1.jar:?]
at com.webank.wedatasphere.linkis.storage.utils.FileSystemUtils$.createNewFile(FileSystemUtils.scala:80) ~[linkis-storage-1.0.0-RC1.jar:?]
at com.webank.wedatasphere.linkis.entrance.log.AbstractLogWriter.<init>(LogWriter.scala:83) ~[linkis-entrance-1.0.0-RC1.jar:?]
at com.webank.wedatasphere.linkis.entrance.log.CacheLogWriter.<init>(CacheLogWriter.scala:31) ~[linkis-entrance-1.0.0-RC1.jar:?]
at com.webank.wedatasphere.linkis.entrance.log.CacheLogManager.createLogWriter(CacheLogManager.scala:62) ~[linkis-entrance-1.0.0-RC1.jar:?]
at com.webank.wedatasphere.linkis.entrance.log.LogManager$$anonfun$onLogUpdate$1.apply$mcV$sp(LogManager.scala:45) ~[linkis-entrance-1.0.0-RC1.jar:?]
at com.webank.wedatasphere.linkis.entrance.log.LogManager$$anonfun$onLogUpdate$1.apply(LogManager.scala:42) ~[linkis-entrance-1.0.0-RC1.jar:?]
at com.webank.wedatasphere.linkis.entrance.log.LogManager$$anonfun$onLogUpdate$1.apply(LogManager.scala:42) ~[linkis-entrance-1.0.0-RC1.jar:?]
at com.webank.wedatasphere.linkis.common.utils.Utils$.tryCatch(Utils.scala:48) [linkis-common-1.0.0-RC1.jar:?]
at com.webank.wedatasphere.linkis.entrance.log.LogManager.onLogUpdate(LogManager.scala:54) [linkis-entrance-1.0.0-RC1.jar:?]
at com.webank.wedatasphere.linkis.entrance.execute.EntranceJob$$anonfun$afterStateChanged$6.apply(EntranceJob.scala:141) [linkis-entrance-1.0.0-RC1.jar:?]
at com.webank.wedatasphere.linkis.entrance.execute.EntranceJob$$anonfun$afterStateChanged$6.apply(EntranceJob.scala:141) [linkis-entrance-1.0.0-RC1.jar:?]
at scala.Option.foreach(Option.scala:257) [scala-library-2.11.12.jar:?]
at com.webank.wedatasphere.linkis.entrance.execute.EntranceJob.afterStateChanged(EntranceJob.scala:141) [linkis-entrance-1.0.0-RC1.jar:?]
at com.webank.wedatasphere.linkis.scheduler.queue.SchedulerEvent$class.transition(SchedulerEvent.scala:82) [linkis-scheduler-1.0.0-RC1.jar:?]
at com.webank.wedatasphere.linkis.scheduler.queue.Job.transition(Job.scala:38) [linkis-scheduler-1.0.0-RC1.jar:?]
at com.webank.wedatasphere.linkis.scheduler.queue.Job$$anonfun$transitionCompleted$3.apply$mcV$sp(Job.scala:215) [linkis-scheduler-1.0.0-RC1.jar:?]
at com.webank.wedatasphere.linkis.scheduler.queue.Job$$anonfun$transitionCompleted$3.apply(Job.scala:215) [linkis-scheduler-1.0.0-RC1.jar:?]
at com.webank.wedatasphere.linkis.scheduler.queue.Job$$anonfun$transitionCompleted$3.apply(Job.scala:215) [linkis-scheduler-1.0.0-RC1.jar:?]
at com.webank.wedatasphere.linkis.common.utils.Utils$.tryCatch(Utils.scala:48) [linkis-common-1.0.0-RC1.jar:?]
at com.webank.wedatasphere.linkis.common.utils.Utils$.tryAndWarnMsg(Utils.scala:88) [linkis-common-1.0.0-RC1.jar:?]
at com.webank.wedatasphere.linkis.scheduler.queue.Job.transitionCompleted(Job.scala:215) [linkis-scheduler-1.0.0-RC1.jar:?]
at com.webank.wedatasphere.linkis.entrance.execute.EntranceJob.transitionCompleted(EntranceJob.scala:164) [linkis-entrance-1.0.0-RC1.jar:?]
at com.webank.wedatasphere.linkis.scheduler.queue.Job.onFailure(Job.scala:117) [linkis-scheduler-1.0.0-RC1.jar:?]
at com.webank.wedatasphere.linkis.entrance.execute.EntranceJob.onFailure(EntranceJob.scala:154) [linkis-entrance-1.0.0-RC1.jar:?]
at com.webank.wedatasphere.linkis.scheduler.queue.fifoqueue.FIFOUserConsumer$$anonfun$loop$3$$anonfun$apply$10.apply(FIFOUserConsumer.scala:150) [linkis-scheduler-1.0.0-RC1.jar:?]
at com.webank.wedatasphere.linkis.scheduler.queue.fifoqueue.FIFOUserConsumer$$anonfun$loop$3$$anonfun$apply$10.apply(FIFOUserConsumer.scala:144) [linkis-scheduler-1.0.0-RC1.jar:?]
at com.webank.wedatasphere.linkis.common.utils.Utils$.tryCatch(Utils.scala:54) [linkis-common-1.0.0-RC1.jar:?]
at com.webank.wedatasphere.linkis.scheduler.queue.fifoqueue.FIFOUserConsumer$$anonfun$loop$3.apply(FIFOUserConsumer.scala:144) [linkis-scheduler-1.0.0-RC1.jar:?]
at com.webank.wedatasphere.linkis.scheduler.queue.fifoqueue.FIFOUserConsumer$$anonfun$loop$3.apply(FIFOUserConsumer.scala:117) [linkis-scheduler-1.0.0-RC1.jar:?]
at scala.Option.foreach(Option.scala:257) [scala-library-2.11.12.jar:?]
at com.webank.wedatasphere.linkis.scheduler.queue.fifoqueue.FIFOUserConsumer.loop(FIFOUserConsumer.scala:117) [linkis-scheduler-1.0.0-RC1.jar:?]
at com.webank.wedatasphere.linkis.scheduler.queue.fifoqueue.FIFOUserConsumer$$anonfun$run$1.apply$mcV$sp(FIFOUserConsumer.scala:82) [linkis-scheduler-1.0.0-RC1.jar:?]
at com.webank.wedatasphere.linkis.scheduler.queue.fifoqueue.FIFOUserConsumer$$anonfun$run$1.apply(FIFOUserConsumer.scala:82) [linkis-scheduler-1.0.0-RC1.jar:?]
at com.webank.wedatasphere.linkis.scheduler.queue.fifoqueue.FIFOUserConsumer$$anonfun$run$1.apply(FIFOUserConsumer.scala:82) [linkis-scheduler-1.0.0-RC1.jar:?]
at com.webank.wedatasphere.linkis.common.utils.Utils$.tryCatch(Utils.scala:48) [linkis-common-1.0.0-RC1.jar:?]
at com.webank.wedatasphere.linkis.common.utils.Utils$.tryAndError(Utils.scala:102) [linkis-common-1.0.0-RC1.jar:?]
at com.webank.wedatasphere.linkis.scheduler.queue.fifoqueue.FIFOUserConsumer.run(FIFOUserConsumer.scala:82) [linkis-scheduler-1.0.0-RC1.jar:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_275]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_275]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_275]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_275]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_275]
Caused by: java.nio.channels.ClosedByInterruptException
at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202) ~[?:1.8.0_275]
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:658) ~[?:1.8.0_275]
at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:192) ~[hadoop-common-2.6.5.jar:?]
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:530) ~[hadoop-common-2.6.5.jar:?]
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:494) ~[hadoop-common-2.6.5.jar:?]
at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:609) ~[hadoop-common-2.6.5.jar:?]
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:707) ~[hadoop-common-2.6.5.jar:?]
at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:370) ~[hadoop-common-2.6.5.jar:?]
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1523) ~[hadoop-common-2.6.5.jar:?]
at org.apache.hadoop.ipc.Client.call(Client.java:1440) ~[hadoop-common-2.6.5.jar:?]
... 61 more
2021-03-30 16:24:29.691 [INFO ] [linkisClient-Test_geosmart_ops_sparkConsumerThread] c.w.w.l.e.j.EntranceExecutionJob (243) [close] - job:linkisClient-Test_geosmart_ops_spark_4 is closing
how to fix this,and I want to know, when does the HDFSFileSystem Close, and when does it open?
when run
com.webank.wedatasphere.linkis.ujes.client.LinkisClientTest
,exception throw when run code
String resultSet = jobInfo.getResultSetList(client)[0]
,the linkis-cg-entrance throw a exception about filesystem;
error debug:
entrance error log
how to fix this,and I want to know, when does the HDFSFileSystem Close, and when does it open?