xuxueli / xxl-job

A distributed task scheduling framework.(分布式任务调度平台XXL-JOB)
http://www.xuxueli.com/xxl-job/
GNU General Public License v3.0
27.62k stars 10.93k forks source link

VERSION 2.0.1 com.xxl.rpc.util.XxlRpcException: java.io.EOFException: HttpConnectionOverHTTP #612

Closed sunhao91 closed 5 years ago

sunhao91 commented 5 years ago

VERSION 2.0.1

有人遇到过下面这张情况么,任务调度的时候,间歇性报错~

半分钟执行一次,查日志,任务也没有堆积,不晓得下面的错误是啥意思

心跳检测: address:192.168.2.44:9999 code:500 msg:com.xxl.rpc.util.XxlRpcException: java.io.EOFException: HttpConnectionOverHTTP@6cb24963(l:/192.168.2.44:2672 <-> r:/192.168.2.44:9999,closed=false)[HttpChannelOverHTTP@52c1a0cb(exchange=HttpExchange@1b1c90ba req=TERMINATED/null@null res=PENDING/null@null)[send=HttpSenderOverHTTP@7bce7ccb(req=QUEUED,snd=COMPLETED,failure=null)[HttpGenerator{s=START}],recv=HttpReceiverOverHTTP@5c9cb80(rsp=IDLE,failure=null)[HttpParser{s=CLOSED,0 of -1}]]] at org.eclipse.jetty.client.http.HttpReceiverOverHTTP.earlyEOF(HttpReceiverOverHTTP.java:277) at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:1305) at org.eclipse.jetty.client.http.HttpReceiverOverHTTP.shutdown(HttpReceiverOverHTTP.java:182) at org.eclipse.jetty.client.http.HttpReceiverOverHTTP.process(HttpReceiverOverHTTP.java:129) at org.eclipse.jetty.client.http.HttpReceiverOverHTTP.receive(HttpReceiverOverHTTP.java:69) at org.eclipse.jetty.client.http.HttpChannelOverHTTP.receive(HttpChannelOverHTTP.java:90) at org.eclipse.jetty.client.http.HttpConnectionOverHTTP.onFillable(HttpConnectionOverHTTP.java:174) at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) at java.lang.Thread.run(Thread.java:748)

RoonyLi commented 5 years ago

jetty 丫经常EOF 看下你的jetty到底是推荐版本吧。

luo1993hao commented 5 years ago

我也碰到这个问题了。有解决思路以及方案吗

soenter commented 5 years ago

遇到了,应该是bug

sunhao91 commented 5 years ago

@luo1993hao

我也碰到这个问题了。有解决思路以及方案吗

解决思路和方案是。。。 我把半分钟执行一次的需求停了。O(∩_∩)O 目前没这需求,我就没跟着这事~ 可以 参考 下@RoonyLi 的建议,更换下jetty 试试 ,我待会找个测试demo更下试试

sunhao91 commented 5 years ago

特意起项目看了下,依赖的就是推荐的 9.2.26.v20180806 版本

ileeoyo commented 5 years ago

发现个问题,项目依赖xxl-core2.0.1打的包,然后版本号默认取了jetty最高版本,而不是xxl-job总pom下的9.2.26.v20180806</jetty-server.version>。真奇怪。版本号不对所以有这个问题吧

dongzehong commented 5 years ago

不知道有没有解决方案了?直接部署的xxl-job-admin,也会有这个错。jetty版本是9.2.26.v20180806。另外xxl-job-admin集群部署的时候,这个错误频率更高。

magicFire commented 5 years ago

我也遇到这个问题了,半分钟执行一次,我这边出现这个EOF报错比较频繁,6%左右。目前看都是调度失败,执行器那边没有收到调度请求。这个报错频率偏高,我都不太敢上线上用了。

xuxueli commented 5 years ago

建议采用Release版本,并且Jetty版本同core包的 ”jetty-server.version“ 保持一致,调度中心与执行器均一致。

最新迭代版本已经废弃Jetty,通讯层默认选型Netty,将会跟随下个版本一同推出,可以体验下。

11003843 commented 5 years ago

触发调度: address:127.0.0.1:9999 code:500 msg:com.xxl.rpc.util.XxlRpcException: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: no further information: /127.0.0.1:9999 at com.xxl.rpc.remoting.invoker.reference.XxlRpcReferenceBean$1.invoke(XxlRpcReferenceBean.java:227) at com.sun.proxy.$Proxy82.run(Unknown Source) at com.xxl.job.admin.core.trigger.XxlJobTrigger.runExecutor(XxlJobTrigger.java:188) at com.xxl.job.admin.core.trigger.XxlJobTrigger.processTrigger(XxlJobTrigger.java:141) at com.xxl.job.admin.core.trigger.XxlJobTrigger.trigger(XxlJobTrigger.java:75) at com.xxl.job.admin.core.thread.JobTriggerPoolHelper$2.run(JobTriggerPoolHelper.java:42) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: no further information: /127.0.0.1:9999 at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:665) at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:612) at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:529) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:491) at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ... 1 more Caused by: java.net.ConnectException: Connection refused: no further information ... 11 more

11003843 commented 5 years ago

请问下:这个netty连接报错,要在哪里设置一下呢?谢谢! xxl-job-core 版本号:2.0.0

codingman1990 commented 5 years ago

@sunhao91 @luo1993hao 我也遇到这个问题,单机部署没有问题,集群就会出现,请问你们是怎么解决的?

codingman1990 commented 5 years ago

@xuxueli 我现在是通过在XxlJobTrigger.runExecutor方法中捕捉异常的地方特殊判断是否msg包含"java.io.EOFException"的XxlRpcException,如果是的话,我再手动触发一次,可以解决这个问题,但是不知道有没有隐患

           // 异常信息文本,由XxlRpcReferenceBean得知直接使用的new XxlRpcException(String msg)构造方法
            String msg = e.getMessage();
            logger.error("测试异常:{}", e.getMessage());
            //logger.error("测试e类型:{},cause类型:{}", e.getClass(), Optional.ofNullable(e.getCause()).map(Throwable::getClass).orElse(null));
            // EOFException特殊处理再触发一次
            String eof = "java.io.EOFException";
            if (e instanceof XxlRpcException && msg.contains(eof)) {
                try {
                    logger.info(">>>>>>>>>>> xxl-job trigger EOFException, auto try again!");
                    runResult = XxlJobDynamicScheduler.getExecutorBiz(address).run(triggerParam);
                } catch (Exception ex) {
                    logger.error(">>>>>>>>>>> xxl-job trigger error, please check if the executor[{}] is running.", address, ex);
                    runResult = new ReturnT<String>(ReturnT.FAIL_CODE, ThrowableUtil.toString(ex));
                }
            } else {
                logger.error(">>>>>>>>>>> xxl-job trigger error, please check if the executor[{}] is running.", address, e);
                runResult = new ReturnT<String>(ReturnT.FAIL_CODE, ThrowableUtil.toString(e));
            }
quintinguo commented 5 years ago

XxlRpcException: java.lang.InterruptedException 我也遇到类似的问题,解决方案如dongzehong所说,更新jetty依赖的版本,我用的xxl-job-core版本是2.0.1,这个版本的jetty依赖9.4.8.v20171121,所以需要从xxl-job-core依赖中排除jetty,然后再重新引入,如下

    <dependency>
        <groupId>com.xuxueli</groupId>
        <artifactId>xxl-job-core</artifactId>
        <version>${xxljob.version}</version>
        <exclusions>
            <exclusion>
                <groupId>org.eclipse.jetty</groupId>
                <artifactId>jetty-client</artifactId>
            </exclusion>
        </exclusions>
    </dependency>
    <dependency>
        <groupId>org.eclipse.jetty</groupId>
        <artifactId>jetty-server</artifactId>
        <version>${jetty.version}</version>
    </dependency>
    <dependency>
        <groupId>org.eclipse.jetty</groupId>
        <artifactId>jetty-client</artifactId>
        <version>${jetty.version}</version>
    </dependency>
    <!--end of xxl-job -->

.......

2.0.1 9.2.26.v20180806