Dubbo2.7.6 没有优雅停机

[shuchaang]

• [ ] I have searched the issues of this repository and believe that this is not a duplicate.
• [ ] I have checked the FAQ of this repository and believe that this is not a duplicate.

Environment

• Dubbo version: 2.7.6
• Operating System version: centos7
• Java version: jdk1.8

Steps to reproduce this issue

1. provider/consumer 都是2.7.6版本 spring版本4.3.2
2. provider配置 -Ddubbo.shutdown.hook=true
3. 服务停止脚本使用: kill [pid]

Expected Result

能够正常优雅停机

Actual Result

每次生产环境服务发布时consumer会收到各种报错信息，貌似dubbo没有做到优雅停机

错误1:

org.apache.dubbo.rpc.RpcException: java.util.concurrent.ExecutionException: java.lang.IllegalStateException: Consumer is shutting down and this call is going to be stopped without receiving any result, usually this is called by a slow provider instance or bad service implementation.

org.apache.dubbo.rpc.AsyncRpcResult.getAppResponse(AsyncRpcResult.java:151)

org.apache.dubbo.rpc.AsyncRpcResult.recreate(AsyncRpcResult.java:191)

org.apache.dubbo.rpc.proxy.InvokerInvocationHandler.invoke(InvokerInvocationHandler.java:74)

org.apache.dubbo.common.bytecode.proxy11.getDetailInfo(proxy11.java)

com.xxx.uc.token.gateway.controller.AuthByTokenController.detailInfo(AuthByTokenController.java:185)

com.xxx.uc.token.gateway.controller.AuthByTokenController$$FastClassBySpringCGLIB$$28c43dc2.invoke()

org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)

org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:720)

org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)

org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:97)

com.xxx.uc.token.gateway.aop.RouteHandlerAOP.route(RouteHandlerAOP.java:119)

com.xxx.uc.token.gateway.aop.RouteHandlerAOP.around(RouteHandlerAOP.java:74)

sun.reflect.GeneratedMethodAccessor426.invoke(Unknown Source)

sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

java.lang.reflect.Method.invoke(Method.java:498)

org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:629)

org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:618)

org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:70)

org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)

org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:97)

com.xxx.pay.commons.service.log.LogAspect.aspect(LogAspect.java:39)

sun.reflect.GeneratedMethodAccessor2386.invoke(Unknown Source)

sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

java.lang.reflect.Method.invoke(Method.java:498)

org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:629)

org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:618)

org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:70)

org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)

org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)

org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)

org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:655)

com.xxx.uc.token.gateway.controller.AuthByTokenController$$EnhancerBySpringCGLIB$$5cc5d2fd.detailInfo()

sun.reflect.GeneratedMethodAccessor2322.invoke(Unknown Source)

sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

java.lang.reflect.Method.invoke(Method.java:498)

org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:221)

org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:136)

org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:114)

org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827)

org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738)

org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)

org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:963)

org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:897)

org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)

org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:872)

javax.servlet.http.HttpServlet.service(HttpServlet.java:661)

org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)

javax.servlet.http.HttpServlet.service(HttpServlet.java:742)

org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)

org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)

org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)

org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)

org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)

com.xxx.usercenter.http.filter.SpringMVCRequestContextFilter.doFilter(SpringMVCRequestContextFilter.java:63)

org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)

org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)

com.xxx.uc.token.gateway.trace.TraceFilter.doFilter(TraceFilter.java:126)

org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)

org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)

com.xxx.usercenter.http.filter.AuthIpVisitUrlFilter.doFilter(AuthIpVisitUrlFilter.java:66)

org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:346)

org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:262)

org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)

org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)

org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:197)

org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)

org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)

org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)

org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:199)

org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)

org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:478)

org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140)

org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81)

org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:650)

org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)

org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342)

org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:803)

org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)

org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:868)

org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1459)

org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)

java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)

java.lang.Thread.run(Thread.java:748)

Caused by: java.util.concurrent.ExecutionException: java.lang.IllegalStateException: Consumer is shutting down and this call is going to be stopped without receiving any result, usually this is called by a slow provider instance or bad service implementation.

java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)

java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908)

org.apache.dubbo.rpc.AsyncRpcResult.getAppResponse(AsyncRpcResult.java:146)

org.apache.dubbo.rpc.AsyncRpcResult.recreate(AsyncRpcResult.java:191)

org.apache.dubbo.rpc.proxy.InvokerInvocationHandler.invoke(InvokerInvocationHandler.java:74)

org.apache.dubbo.common.bytecode.proxy11.getDetailInfo(proxy11.java)

com.xxx.uc.token.gateway.controller.AuthByTokenController.detailInfo(AuthByTokenController.java:185)

com.xxx.uc.token.gateway.controller.AuthByTokenController$$FastClassBySpringCGLIB$$28c43dc2.invoke()

org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)

org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:720)

org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)

org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:97)

com.xxx.uc.token.gateway.aop.RouteHandlerAOP.route(RouteHandlerAOP.java:119)

com.xxx.uc.token.gateway.aop.RouteHandlerAOP.around(RouteHandlerAOP.java:74)

sun.reflect.GeneratedMethodAccessor426.invoke(Unknown Source)

sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

java.lang.reflect.Method.invoke(Method.java:498)

org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:629)

org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:618)

org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:70)

org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)

org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:97)

com.xxx.pay.commons.service.log.LogAspect.aspect(LogAspect.java:39)

sun.reflect.GeneratedMethodAccessor2386.invoke(Unknown Source)

sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

java.lang.reflect.Method.invoke(Method.java:498)

org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:629)

org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:618)

org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:70)

org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)

org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)

org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)

org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:655)

com.xxx.uc.token.gateway.controller.AuthByTokenController$$EnhancerBySpringCGLIB$$5cc5d2fd.detailInfo()

sun.reflect.GeneratedMethodAccessor2322.invoke(Unknown Source)

sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

java.lang.reflect.Method.invoke(Method.java:498)

org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:221)

org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:136)

org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:114)

org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827)

org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738)

org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)

org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:963)

org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:897)

org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)

org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:872)

javax.servlet.http.HttpServlet.service(HttpServlet.java:661)

org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)

javax.servlet.http.HttpServlet.service(HttpServlet.java:742)

org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)

org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)

org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)

org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)

org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)

com.xxx.usercenter.http.filter.SpringMVCRequestContextFilter.doFilter(SpringMVCRequestContextFilter.java:63)

org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)

org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)

com.xxx.uc.token.gateway.trace.TraceFilter.doFilter(TraceFilter.java:126)

org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)

org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)

com.xxx.usercenter.http.filter.AuthIpVisitUrlFilter.doFilter(AuthIpVisitUrlFilter.java:66)

org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:346)

org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:262)

org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)

org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)

org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:197)

org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)

org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)

org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)

org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:199)

org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)

org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:478)

org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140)

org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81)

org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:650)

org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)

org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342)

org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:803)

org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)

org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:868)

org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1459)

org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)

java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)

java.lang.Thread.run(Thread.java:748)

Caused by: java.lang.IllegalStateException: Consumer is shutting down and this call is going to be stopped without receiving any result, usually this is called by a slow provider instance or bad service implementation.

org.apache.dubbo.common.threadpool.ThreadlessExecutor.shutdownNow(ThreadlessExecutor.java:164)

org.apache.dubbo.remoting.exchange.support.DefaultFuture.closeChannel(DefaultFuture.java:147)

org.apache.dubbo.remoting.exchange.support.header.HeaderExchangeHandler.disconnected(HeaderExchangeHandler.java:134)

org.apache.dubbo.remoting.transport.AbstractChannelHandlerDelegate.disconnected(AbstractChannelHandlerDelegate.java:48)

org.apache.dubbo.remoting.transport.dispatcher.ChannelEventRunnable.run(ChannelEventRunnable.java:73)

java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

java.lang.Thread.run(Thread.java:748)

错误2:

[DUBBO] Got exception when trying to fetch the underlying result from asyncRpcResult,dubbo version:2.7.6

错误3:

consumer 会接收到NPE,原因是执行kill 之后, dubbo引用到了已经销毁的spring bean

dubbo的启动类是自己封装的 调用了一些前置的方法:

public class Main{

     public static void main(String[] args){
               ///执行一些方法
             org.apache.dubbo.container.Main.main(args)
}

[xiaoheng1]

relation https://github.com/apache/dubbo/issues/7093

[xiaoheng1]

&READY-TO-CLOSE&

[shuchaang]

dubbo 2.7.6 在多注册中心环境下有bug,2.7.7已经解决