weibocom / motan

A cross-language remote procedure call(RPC) framework for rapid development of high performance distributed services.
Other
5.89k stars 1.78k forks source link

负载均衡策略localFirst服务调用,接口中某个方法突然调不到了,接口里的其他方法都正常调用 #650

Closed Python-IBM closed 6 years ago

Python-IBM commented 6 years ago

motan版本用最新的1.1.0,用的是motan-transport-netty4,client请求超时requestTimeout设置为20000,麻烦官方技术人员帮忙看看是什么问题

2018-01-30 17:15:05,738 INFO com.weibo.api.motan.log.DefaultLogService.accessLog(DefaultLogService.java:91) - referer|app-server|192.168.10.3|com.baobao.service.PointService|searchPointsByCityCode|java.lang.String%2Cjava.lang.String%2Cjava.lang.Long%2Cjava.lang.String%2Cjava.lang.String|192.168.10.6|app-server|false|20000 2018-01-30 17:15:05,738 ERROR com.weibo.api.motan.log.DefaultLogService.error(DefaultLogService.java:87) - RefererInvocationHandler invoke Error: uri=motan://192.168.10.3:0/com.baobao.service.PointService requestId=1591008229576004909 interface=com.baobao.service.PointService method=searchPointsByCityCode(java.lang.String,java.lang.String,java.lang.Long,java.lang.String,java.lang.String) com.weibo.api.motan.exception.MotanServiceException: error_message: com.weibo.api.motan.rpc.DefaultResponseFuture request timeout: serverPort=192.168.10.6:8601 requestId=1591008229576004909 interface=com.baobao.service.PointService method=searchPointsByCityCode(java.lang.String,java.lang.String,java.lang.Long,java.lang.String,java.lang.String) cost=20000, status: 503, error_code: 10003,r=null at com.weibo.api.motan.rpc.DefaultResponseFuture.timeoutSoCancel(DefaultResponseFuture.java:217) at com.weibo.api.motan.rpc.DefaultResponseFuture.getValue(DefaultResponseFuture.java:115) at com.weibo.api.motan.rpc.DefaultResponse.(DefaultResponse.java:54) at com.weibo.api.motan.transport.netty4.NettyClient.asyncResponse(NettyClient.java:138) at com.weibo.api.motan.transport.netty4.NettyClient.request(NettyClient.java:122) at com.weibo.api.motan.transport.netty4.NettyClient.request(NettyClient.java:76) at com.weibo.api.motan.protocol.rpc.DefaultRpcReferer.doCall(DefaultRpcReferer.java:52) at com.weibo.api.motan.rpc.AbstractReferer.call(AbstractReferer.java:64) at com.weibo.api.motan.filter.AccessLogFilter.filter(AccessLogFilter.java:56) at com.weibo.api.motan.protocol.support.ProtocolFilterDecorator$1.call(ProtocolFilterDecorator.java:86) at com.weibo.api.motan.cluster.ha.FailfastHaStrategy.call(FailfastHaStrategy.java:38) at com.weibo.api.motan.cluster.support.ClusterSpi.call(ClusterSpi.java:73) at com.weibo.api.motan.proxy.RefererInvocationHandler.invoke(RefererInvocationHandler.java:132) at com.sun.proxy.$Proxy182.searchPointsByCityCode(Unknown Source) at com.baobao.app.controller.PointsController.searchPointsByCityCode(PointsController.java:60) at sun.reflect.GeneratedMethodAccessor710.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:483) at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205) at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133) at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97) at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827) at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738) at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85) at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967) at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901) at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970) at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:872) at javax.servlet.http.HttpServlet.service(HttpServlet.java:661) at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846) at javax.servlet.http.HttpServlet.service(HttpServlet.java:742) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.springframework.web.filter.HttpPutFormContentFilter.doFilterInternal(HttpPutFormContentFilter.java:108) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:81) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:197) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.springframework.boot.web.support.ErrorPageFilter.doFilter(ErrorPageFilter.java:115) at org.springframework.boot.web.support.ErrorPageFilter.access$000(ErrorPageFilter.java:59) at org.springframework.boot.web.support.ErrorPageFilter$1.doFilterInternal(ErrorPageFilter.java:90) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) at org.springframework.boot.web.support.ErrorPageFilter.doFilter(ErrorPageFilter.java:108) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:199) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:478) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81) at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:650) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342) at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:803) at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66) at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:868) at org.apache.tomcat.util.net.Nio2Endpoint$SocketProcessor.doRun(Nio2Endpoint.java:1693) at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.lang.Thread.run(Thread.java:744) 2018-01-30 17:15:05,738 ERROR com.baobao.app.handler.GlobalHandler.exception(GlobalHandler.java:27) - 程序异常 com.weibo.api.motan.exception.MotanServiceException: error_message: com.weibo.api.motan.rpc.DefaultResponseFuture request timeout: serverPort=192.168.10.6:8601 requestId=1591008229576004909 interface=com.baobao.service.PointService method=searchPointsByCityCode(java.lang.String,java.lang.String,java.lang.Long,java.lang.String,java.lang.String) cost=20000, status: 503, error_code: 10003,r=null at com.weibo.api.motan.rpc.DefaultResponseFuture.timeoutSoCancel(DefaultResponseFuture.java:217) at com.weibo.api.motan.rpc.DefaultResponseFuture.getValue(DefaultResponseFuture.java:115) at com.weibo.api.motan.rpc.DefaultResponse.(DefaultResponse.java:54) at com.weibo.api.motan.transport.netty4.NettyClient.asyncResponse(NettyClient.java:138) at com.weibo.api.motan.transport.netty4.NettyClient.request(NettyClient.java:122) at com.weibo.api.motan.transport.netty4.NettyClient.request(NettyClient.java:76) at com.weibo.api.motan.protocol.rpc.DefaultRpcReferer.doCall(DefaultRpcReferer.java:52) at com.weibo.api.motan.rpc.AbstractReferer.call(AbstractReferer.java:64) at com.weibo.api.motan.filter.AccessLogFilter.filter(AccessLogFilter.java:56) at com.weibo.api.motan.protocol.support.ProtocolFilterDecorator$1.call(ProtocolFilterDecorator.java:86) at com.weibo.api.motan.cluster.ha.FailfastHaStrategy.call(FailfastHaStrategy.java:38) at com.weibo.api.motan.cluster.support.ClusterSpi.call(ClusterSpi.java:73) at com.weibo.api.motan.proxy.RefererInvocationHandler.invoke(RefererInvocationHandler.java:132) at com.sun.proxy.$Proxy182.searchPointsByCityCode(Unknown Source) at com.baobao.app.controller.PointsController.searchPointsByCityCode(PointsController.java:60) at sun.reflect.GeneratedMethodAccessor710.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:483) at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205) at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133) at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97) at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827) at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738) at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85) at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967) at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901) at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970) at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:872) at javax.servlet.http.HttpServlet.service(HttpServlet.java:661) at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846) at javax.servlet.http.HttpServlet.service(HttpServlet.java:742) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.springframework.web.filter.HttpPutFormContentFilter.doFilterInternal(HttpPutFormContentFilter.java:108) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:81) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:197) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.springframework.boot.web.support.ErrorPageFilter.doFilter(ErrorPageFilter.java:115) at org.springframework.boot.web.support.ErrorPageFilter.access$000(ErrorPageFilter.java:59) at org.springframework.boot.web.support.ErrorPageFilter$1.doFilterInternal(ErrorPageFilter.java:90) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) at org.springframework.boot.web.support.ErrorPageFilter.doFilter(ErrorPageFilter.java:108) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:199) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:478) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81) at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:650) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342) at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:803) at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66) at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:868) at org.apache.tomcat.util.net.Nio2Endpoint$SocketProcessor.doRun(Nio2Endpoint.java:1693) at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.lang.Thread.run(Thread.java:744)

rayzhang0603 commented 6 years ago

从异常看是20s没有获取到返回值超时了。motan的请求是按channel处理的,不区分method,不会出现只有某个方法请求失败的问题。

你可以在确认一下,异常发生时,这个service下所有的方法都超时,还是只有这一个方法超时,如果只有这一个方法超时,建议看一下server端这个方法的执行时间是否正常。

另外可以看一下client端或server端有没有什么可疑的error日志,特别是编解码异常,编解码异常,反应在请求上会是超时。

Python-IBM commented 6 years ago

好的,非常感谢!我们在自己看下。