search

apache / dubbo

The java implementation of Apache Dubbo. An RPC and microservice framework.
https://dubbo.apache.org/
Apache License 2.0
40.48k stars 26.43k forks source link

NoClassDefFoundError during graceful shutdown when using tomcat #1474

Closed guoxiangdaxia closed 6 years ago

guoxiangdaxia commented 6 years ago
  1. provider和consumer都设置-Ddubbo.service.shutdown.wait=120000。
  2. 同样的配置用jetty就没问题，用tomcat就出现了以下问题。
  3. 请帮忙看下，不胜感激。 provider日志： service--进入--b [ztyq_test-service] [2018-03-16 15:54:00:904] [INFO] - com.alibaba.dubbo.config.AbstractConfig(80) - [DUBBO] Run shutdown hook now., dubbo version: 2.5.6, current host: 127.0.0.1 [ztyq_test-service] [2018-03-16 15:54:00:905] [INFO] - com.alibaba.dubbo.registry.support.AbstractRegistryFactory(64) - [DUBBO] Close all registries [zookeeper://localhost:2181/com.alibaba.dubbo.registry.RegistryService?application=ztyq_test-service&client=curator&dubbo=2.5.6&file=${user.home}/output/dubbo.cache&interface=com.alibaba.dubbo.registry.RegistryService&pid=6664&timestamp=1521186655106], dubbo version: 2.5.6, current host: 127.0.0.1 [ztyq_test-service] [2018-03-16 15:54:00:905] [INFO] - com.alibaba.dubbo.registry.zookeeper.ZookeeperRegistry(471) - [DUBBO] Destroy registry:zookeeper://localhost:2181/com.alibaba.dubbo.registry.RegistryService?application=ztyq_test-service&client=curator&dubbo=2.5.6&file=${user.home}/output/dubbo.cache&interface=com.alibaba.dubbo.registry.RegistryService&pid=6664&timestamp=1521186655106, dubbo version: 2.5.6, current host: 127.0.0.1 [ztyq_test-service] [2018-03-16 15:54:00:905] [INFO] - com.alibaba.dubbo.registry.zookeeper.ZookeeperRegistry(480) - [DUBBO] Destroy unregister url dubbo://10.255.1.94:20885/com.test.ztyq.service.BaseService?anyhost=true&application=ztyq_test-service&default.loadbalance=leastactive&default.retries=0&default.timeout=50000&dubbo=2.5.6&generic=false&interface=com.test.ztyq.service.BaseService&methods=getEncryptStr,getDecryptStr&pid=6664&serialization=hessian2&side=provider&timestamp=1521186655116, dubbo version: 2.5.6, current host: 127.0.0.1 [ztyq_test-service] [2018-03-16 15:54:00:905] [INFO] - com.alibaba.dubbo.registry.zookeeper.ZookeeperRegistry(496) - [DUBBO] Destroy unsubscribe url provider://10.255.1.94:20885/com.test.ztyq.service.BaseService?anyhost=true&application=ztyq_test-service&category=configurators&check=false&default.loadbalance=leastactive&default.retries=0&default.timeout=50000&dubbo=2.5.6&generic=false&interface=com.test.ztyq.service.BaseService&methods=getEncryptStr,getDecryptStr&pid=6664&serialization=hessian2&side=provider&timestamp=1521186655116, dubbo version: 2.5.6, current host: 127.0.0.1 [ztyq_test-service] [2018-03-16 15:54:00:905] [DEBUG] - org.apache.curator.framework.imps.CuratorFrameworkImpl(282) - Closing [ztyq_test-service] [2018-03-16 15:54:00:907] [INFO] - org.apache.curator.framework.imps.CuratorFrameworkImpl(821) - backgroundOperationsLoop exiting [ztyq_test-service] [2018-03-16 15:54:00:908] [DEBUG] - org.apache.curator.CuratorZookeeperClient(199) - Closing [ztyq_test-service] [2018-03-16 15:54:00:908] [DEBUG] - org.apache.curator.ConnectionState(115) - Closing [ztyq_test-service] [2018-03-16 15:54:00:915] [DEBUG] - org.apache.zookeeper.ZooKeeper(673) - Closing session: 0x1622c6cc96d0063 [ztyq_test-service] [2018-03-16 15:54:00:915] [DEBUG] - org.apache.zookeeper.ClientCnxn(1370) - Closing client for session: 0x1622c6cc96d0063 [ztyq_test-service] [2018-03-16 15:54:00:918] [DEBUG] - org.apache.zookeeper.ClientCnxn(843) - Reading reply sessionid:0x1622c6cc96d0063, packet:: clientPath:null serverPath:null finished:false header:: 9,-11 replyHeader:: 9,919958,0 request:: null response:: null [ztyq_test-service] [2018-03-16 15:54:00:918] [DEBUG] - org.apache.zookeeper.ClientCnxn(1354) - Disconnecting client for session: 0x1622c6cc96d0063 [ztyq_test-service] [2018-03-16 15:54:00:919] [DEBUG] - org.apache.zookeeper.ClientCnxn(1146) - An exception was thrown while closing send thread for session 0x1622c6cc96d0063 : Unable to read additional data from server sessionid 0x1622c6cc96d0063, likely server has closed socket [ztyq_test-service] [2018-03-16 15:54:00:919] [INFO] - org.apache.zookeeper.ZooKeeper(684) - Session: 0x1622c6cc96d0063 closed [ztyq_test-service] [2018-03-16 15:54:00:929] [INFO] - org.apache.zookeeper.ClientCnxn(519) - EventThread shut down for session: 0x1622c6cc96d0063 [ztyq_test-service] [2018-03-16 15:54:00:929] [INFO] - com.alibaba.dubbo.rpc.protocol.dubbo.DubboProtocol(382) - [DUBBO] Close dubbo server: /10.255.1.94:20885, dubbo version: 2.5.6, current host: 127.0.0.1 service--返回

consumer日志： 进入web--test--b [ztyq_test-web] [2018-03-16 15:54:00:921] [INFO] - com.alibaba.dubbo.registry.zookeeper.ZookeeperRegistry(404) - [DUBBO] Notify urls for subscribe url consumer://10.255.1.94/com.test.ztyq.service.BaseService?application=ztyq_test-web&category=providers,configurators,routers&default.check=false&dubbo=2.5.6&interface=com.test.ztyq.service.BaseService&methods=getEncryptStr,getDecryptStr&pid=10008&side=consumer&timeout=40000&timestamp=1521186341932, urls: [empty://10.255.1.94/com.test.ztyq.service.BaseService?application=ztyq_test-web&category=providers&default.check=false&dubbo=2.5.6&interface=com.test.ztyq.service.BaseService&methods=getEncryptStr,getDecryptStr&pid=10008&side=consumer&timeout=40000&timestamp=1521186341932], dubbo version: 2.5.6, current host: 10.255.1.94 com.alibaba.dubbo.rpc.RpcException: Failed to invoke the method getEncryptStr in the service com.test.ztyq.service.BaseService. Tried 1 times of the providers [10.255.1.94:20885] (1/1) from the registry localhost:2181 on the consumer 10.255.1.94 using the dubbo version 2.5.6. Last error is: Failed to invoke remote method: getEncryptStr, provider: dubbo://10.255.1.94:20885/com.test.ztyq.service.BaseService?anyhost=true&application=ztyq_test-web&check=false&default.check=false&default.loadbalance=leastactive&default.retries=0&default.timeout=50000&dubbo=2.5.6&generic=false&interface=com.test.ztyq.service.BaseService&methods=getEncryptStr,getDecryptStr&pid=10008&remote.timestamp=1521186655116&serialization=hessian2&side=consumer&timeout=40000&timestamp=1521186341932, cause: Failed to send response: Response [id=4, version=2.0.0, status=20, event=false, error=null, result=RpcResult [result=com.test.ztyq.entity.ApiAccResData@1fc39826, exception=null]], cause: java.lang.NoClassDefFoundError: com/alibaba/com/caucho/hessian/io/JavaSerializer java.lang.NoClassDefFoundError: com/alibaba/com/caucho/hessian/io/JavaSerializer at com.alibaba.com.caucho.hessian.io.SerializerFactory.getSerializer(SerializerFactory.java:326) at com.alibaba.com.caucho.hessian.io.Hessian2Output.writeObject(Hessian2Output.java:381) at com.alibaba.dubbo.common.serialize.support.hessian.Hessian2ObjectOutput.writeObject(Hessian2ObjectOutput.java:79) at com.alibaba.dubbo.rpc.protocol.dubbo.DubboCodec.encodeResponseData(DubboCodec.java:193) at com.alibaba.dubbo.remoting.exchange.codec.ExchangeCodec.encodeResponse(ExchangeCodec.java:269) at com.alibaba.dubbo.remoting.exchange.codec.ExchangeCodec.encode(ExchangeCodec.java:70) at com.alibaba.dubbo.rpc.protocol.dubbo.DubboCountCodec.encode(DubboCountCodec.java:39) at com.alibaba.dubbo.remoting.transport.netty.NettyCodecAdapter$InternalEncoder.encode(NettyCodecAdapter.java:81) at org.jboss.netty.handler.codec.oneone.OneToOneEncoder.handleDownstream(OneToOneEncoder.java:66) at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:591) at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:776) at org.jboss.netty.channel.SimpleChannelHandler.writeRequested(SimpleChannelHandler.java:304) at com.alibaba.dubbo.remoting.transport.netty.NettyHandler.writeRequested(NettyHandler.java:99) at org.jboss.netty.channel.SimpleChannelHandler.handleDownstream(SimpleChannelHandler.java:266) at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:591) at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:582) at org.jboss.netty.channel.Channels.write(Channels.java:611) at org.jboss.netty.channel.Channels.write(Channels.java:578) at org.jboss.netty.channel.AbstractChannel.write(AbstractChannel.java:251) at com.alibaba.dubbo.remoting.transport.netty.NettyChannel.send(NettyChannel.java:98) at com.alibaba.dubbo.remoting.transport.AbstractPeer.send(AbstractPeer.java:54) at com.alibaba.dubbo.remoting.exchange.support.header.HeaderExchangeHandler.received(HeaderExchangeHandler.java:171) at com.alibaba.dubbo.remoting.transport.DecodeHandler.received(DecodeHandler.java:52) at com.alibaba.dubbo.remoting.transport.dispatcher.ChannelEventRunnable.run(ChannelEventRunnable.java:81) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.ClassNotFoundException: com.alibaba.com.caucho.hessian.io.JavaSerializer at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1858) at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1709) ... 27 more

at com.alibaba.dubbo.rpc.cluster.support.FailoverClusterInvoker.doInvoke(FailoverClusterInvoker.java:101)
at com.alibaba.dubbo.rpc.cluster.support.AbstractClusterInvoker.invoke(AbstractClusterInvoker.java:229)
at com.alibaba.dubbo.rpc.cluster.support.wrapper.MockClusterInvoker.invoke(MockClusterInvoker.java:72)
at com.alibaba.dubbo.rpc.proxy.InvokerInvocationHandler.invoke(InvokerInvocationHandler.java:52)
at com.alibaba.dubbo.common.bytecode.proxy0.getEncryptStr(proxy0.java)
at com.ztyq.controller.ReturnPageController.test(ReturnPageController.java:80)[ztyq_test-web] [2018-03-16 15:54:20:916] [INFO] - com.alibaba.dubbo.rpc.protocol.dubbo.DubboProtocol(122) -  [DUBBO] disconected from /10.255.1.94:20885,url:dubbo://10.255.1.94:20885/com.test.ztyq.service.BaseService?anyhost=true&application=ztyq_test-web&check=false&codec=dubbo&default.check=false&default.loadbalance=leastactive&default.retries=0&default.timeout=50000&dubbo=2.5.6&generic=false&heartbeat=60000&interface=com.test.ztyq.service.BaseService&methods=getEncryptStr,getDecryptStr&pid=10008&remote.timestamp=1521186655116&serialization=hessian2&side=consumer&timeout=40000&timestamp=1521186341932, dubbo version: 2.5.6, current host: 10.255.1.94

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:221)
at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:137)
at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:110)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:806)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:729)
at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:959)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:893)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)
at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:872)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:85)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:326)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
at org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:945)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:756)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:218)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)

Caused by: com.alibaba.dubbo.remoting.RemotingException: Failed to send response: Response [id=4, version=2.0.0, status=20, event=false, error=null, result=RpcResult [result=com.test.ztyq.entity.ApiAccResData@1fc39826, exception=null]], cause: java.lang.NoClassDefFoundError: com/alibaba/com/caucho/hessian/io/JavaSerializer java.lang.NoClassDefFoundError: com/alibaba/com/caucho/hessian/io/JavaSerializer at com.alibaba.com.caucho.hessian.io.SerializerFactory.getSerializer(SerializerFactory.java:326) at com.alibaba.com.caucho.hessian.io.Hessian2Output.writeObject(Hessian2Output.java:381) at com.alibaba.dubbo.common.serialize.support.hessian.Hessian2ObjectOutput.writeObject(Hessian2ObjectOutput.java:79) at com.alibaba.dubbo.rpc.protocol.dubbo.DubboCodec.encodeResponseData(DubboCodec.java:193) at com.alibaba.dubbo.remoting.exchange.codec.ExchangeCodec.encodeResponse(ExchangeCodec.java:269) at com.alibaba.dubbo.remoting.exchange.codec.ExchangeCodec.encode(ExchangeCodec.java:70) at com.alibaba.dubbo.rpc.protocol.dubbo.DubboCountCodec.encode(DubboCountCodec.java:39) at com.alibaba.dubbo.remoting.transport.netty.NettyCodecAdapter$InternalEncoder.encode(NettyCodecAdapter.java:81) at org.jboss.netty.handler.codec.oneone.OneToOneEncoder.handleDownstream(OneToOneEncoder.java:66) at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:591) at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:776) at org.jboss.netty.channel.SimpleChannelHandler.writeRequested(SimpleChannelHandler.java:304) at com.alibaba.dubbo.remoting.transport.netty.NettyHandler.writeRequested(NettyHandler.java:99) at org.jboss.netty.channel.SimpleChannelHandler.handleDownstream(SimpleChannelHandler.java:266) at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:591) at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:582) at org.jboss.netty.channel.Channels.write(Channels.java:611) at org.jboss.netty.channel.Channels.write(Channels.java:578) at org.jboss.netty.channel.AbstractChannel.write(AbstractChannel.java:251) at com.alibaba.dubbo.remoting.transport.netty.NettyChannel.send(NettyChannel.java:98) at com.alibaba.dubbo.remoting.transport.AbstractPeer.send(AbstractPeer.java:54) at com.alibaba.dubbo.remoting.exchange.support.header.HeaderExchangeHandler.received(HeaderExchangeHandler.java:171) at com.alibaba.dubbo.remoting.transport.DecodeHandler.received(DecodeHandler.java:52) at com.alibaba.dubbo.remoting.transport.dispatcher.ChannelEventRunnable.run(ChannelEventRunnable.java:81) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.ClassNotFoundException: com.alibaba.com.caucho.hessian.io.JavaSerializer at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1858) at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1709) ... 27 more

at com.alibaba.dubbo.remoting.exchange.support.DefaultFuture.returnFromResponse(DefaultFuture.java:220)
at com.alibaba.dubbo.remoting.exchange.support.DefaultFuture.get(DefaultFuture.java:139)
at com.alibaba.dubbo.remoting.exchange.support.DefaultFuture.get(DefaultFuture.java:113)
at com.alibaba.dubbo.rpc.protocol.dubbo.DubboInvoker.doInvoke(DubboInvoker.java:97)
at com.alibaba.dubbo.rpc.protocol.AbstractInvoker.invoke(AbstractInvoker.java:144)
at com.alibaba.dubbo.rpc.listener.ListenerInvokerWrapper.invoke(ListenerInvokerWrapper.java:74)
at com.alibaba.dubbo.rpc.protocol.dubbo.filter.FutureFilter.invoke(FutureFilter.java:54)
at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:69)
at com.alibaba.dubbo.monitor.support.MonitorFilter.invoke(MonitorFilter.java:75)
at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:69)
at com.alibaba.dubbo.rpc.filter.ConsumerContextFilter.invoke(ConsumerContextFilter.java:48)
at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:69)
at com.alibaba.dubbo.rpc.protocol.InvokerWrapper.invoke(InvokerWrapper.java:53)
at com.alibaba.dubbo.rpc.cluster.support.FailoverClusterInvoker.doInvoke(FailoverClusterInvoker.java:77)
... 41 more
chickenlj commented 6 years ago

I am a little confused of your stack trace. Why would a consumer throw "Failed to send response" exception. Could you describe the detailed procedure of stopping services?

zonghaishang commented 6 years ago

Caused by: java.lang.ClassNotFoundException: com.alibaba.com.caucho.hessian.io.JavaSerializer

@guoxiangdaxia Check the packaged dubbo.jar(hessian-lite module)， If it is maven download package, delete the local repository and download it again, mvn -U clean install

diecui1202 commented 6 years ago

Without feedback, we will close it. Feel free to reopen if necessary.