search

sofastack / sofa-rpc

SOFARPC is a high-performance, high-extensibility, production-level Java RPC framework.
https://www.sofastack.tech/sofa-rpc/docs/Home
Apache License 2.0
3.81k stars 1.17k forks source link

服务器运行一段时间之后，com.alipay.sofa.rpc.core.exception.SofaRpcException: com.alipay.remoting.exception.ConnectionClosedException #1102

Closed rabbit1013 closed 2 years ago

rabbit1013 commented 2 years ago

Your question

com.alipay.sofa.rpc.core.exception.SofaRpcException: com.alipay.remoting.exception.ConnectionClosedException: Connection closed! the address is 172.17.223.67:12201 at com.alipay.sofa.rpc.transport.bolt.BoltClientTransport.convertToRpcException(BoltClientTransport.java:351) at com.alipay.sofa.rpc.transport.bolt.BoltClientTransport.syncSend(BoltClientTransport.java:255) at com.alipay.sofa.rpc.client.AbstractCluster.doSendMsg(AbstractCluster.java:586) at com.alipay.sofa.rpc.client.AbstractCluster.sendMsg(AbstractCluster.java:557) at com.alipay.sofa.rpc.filter.ConsumerInvoker.invoke(ConsumerInvoker.java:60) at com.alipay.sofa.rpc.filter.PressureMarkTransformFilter.invoke(PressureMarkTransformFilter.java:68) at com.alipay.sofa.rpc.filter.FilterInvoker.invoke(FilterInvoker.java:100) at io.seata.integration.sofa.rpc.TransactionContextConsumerFilter.invoke(TransactionContextConsumerFilter.java:65) at com.alipay.sofa.rpc.filter.FilterInvoker.invoke(FilterInvoker.java:100) at com.alipay.sofa.rpc.filter.sofatracer.ConsumerTracerFilter.invoke(ConsumerTracerFilter.java:66) at com.alipay.sofa.rpc.filter.FilterInvoker.invoke(FilterInvoker.java:100) at com.alipay.sofa.rpc.filter.RpcReferenceContextFilter.invoke(RpcReferenceContextFilter.java:80) at com.alipay.sofa.rpc.filter.FilterInvoker.invoke(FilterInvoker.java:100) at com.alipay.sofa.rpc.filter.ConsumerExceptionFilter.invoke(ConsumerExceptionFilter.java:37) at com.alipay.sofa.rpc.filter.FilterInvoker.invoke(FilterInvoker.java:100) at com.improve.edu.rpc.filter.SofaGlobalExceptionFilter.invoke(SofaGlobalExceptionFilter.java:30) at com.alipay.sofa.rpc.filter.FilterInvoker.invoke(FilterInvoker.java:100) at com.alipay.sofa.rpc.filter.ConsumerCustomHeaderFilter.invoke(ConsumerCustomHeaderFilter.java:42) at com.alipay.sofa.rpc.filter.FilterInvoker.invoke(FilterInvoker.java:100) at com.alipay.sofa.rpc.filter.FilterChain.invoke(FilterChain.java:268) at com.alipay.sofa.rpc.client.AbstractCluster.filterChain(AbstractCluster.java:550) at com.alipay.sofa.rpc.client.FailoverCluster.doInvoke(FailoverCluster.java:68) at com.alipay.sofa.rpc.client.AbstractCluster.invoke(AbstractCluster.java:297) at com.alipay.sofa.rpc.client.ClientProxyInvoker.invoke(ClientProxyInvoker.java:83) at com.improve.edu.rpc.tms.StudentMiniRpcService_proxy_14.testConnection(StudentMiniRpcService_proxy_14.java) at jdk.internal.reflect.GeneratedMethodAccessor399.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at com.alipay.sofa.runtime.service.binding.JvmBindingAdapter$JvmServiceInvoker.doInvoke(JvmBindingAdapter.java:253) at com.alipay.sofa.runtime.spi.service.ServiceProxy.invoke(ServiceProxy.java:40) at com.alipay.sofa.runtime.service.binding.JvmBindingAdapter$JvmServiceInvoker.invoke(JvmBindingAdapter.java:158) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212) at com.sun.proxy.$Proxy147.testConnection(Unknown Source) at com.improve.edu.mobile.controller.LoginController.test(LoginController.java:344) at com.improve.edu.mobile.controller.LoginController$$FastClassBySpringCGLIB$$10fe0e64.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:771) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749) at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:88) at com.improve.edu.log.aop.RequestLogAspect.time(RequestLogAspect.java:44) at jdk.internal.reflect.GeneratedMethodAccessor181.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:644) at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:633) at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:70) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:95) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:691) at com.improve.edu.mobile.controller.LoginController$$EnhancerBySpringCGLIB$$a11d454c.test() at jdk.internal.reflect.GeneratedMethodAccessor398.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:190) at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:138) at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:105) at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:878) at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:792) at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87) at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1040) at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:943) at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006) at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:898) at javax.servlet.http.HttpServlet.service(HttpServlet.java:626) at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883) at javax.servlet.http.HttpServlet.service(HttpServlet.java:733) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:227) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) at com.improve.edu.common.web.filter.log.ContentCachingFilter.doFilter(ContentCachingFilter.java:28) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.doFilterInternal(WebMvcMetricsFilter.java:93) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) at com.alipay.sofa.tracer.plugins.springmvc.SpringMvcSofaTracerFilter.doFilter(SpringMvcSofaTracerFilter.java:87) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:542) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:143) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:346) at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:374) at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65) at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:887) at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1684) at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.base/java.lang.Thread.run(Thread.java:829) Caused by: com.alipay.remoting.exception.ConnectionClosedException: Connection closed! the address is 172.17.223.67:12201 at com.alipay.remoting.rpc.RpcResponseResolver.preProcess(RpcResponseResolver.java:91) at com.alipay.remoting.rpc.RpcResponseResolver.resolveResponseObject(RpcResponseResolver.java:54) at com.alipay.remoting.rpc.RpcRemoting.invokeSync(RpcRemoting.java:186) at com.alipay.remoting.rpc.RpcClientRemoting.invokeSync(RpcClientRemoting.java:64) at com.alipay.remoting.rpc.RpcClient.invokeSync(RpcClient.java:355) at com.alipay.sofa.rpc.transport.bolt.BoltClientTransport.doInvokeSync(BoltClientTransport.java:279) at com.alipay.sofa.rpc.transport.bolt.BoltClientTransport.syncSend(BoltClientTransport.java:252) ... 112 common frames omitted

Your scenes

describe your use scenes (why need this feature)

Your advice

describe the advice or solution you'd like

Environment

leizhiyuan commented 2 years ago

rpc是哪个版本？

会持续性的发心跳。理论上不会存在这个问题的。可以看下logs/bolt目录下面的日志，可以打包发出来。

rabbit1013 commented 2 years ago

rpc是哪个版本？ 5.7.9 sofaboot 是3.8.0
会持续性的发心跳。理论上不会存在这个问题的。可以看下logs/bolt目录下面的日志，可以打包发出来。 很奇怪 他是运行大概一段时间，有时候是一天，有时候有一周，有时候10个小时就出现这个问题

rabbit1013 commented 2 years ago

这个是连接事件得日志

2021-09-27 23:57:40,631 INFO [ConnectionEventHandler#297] [bolt-netty-client-worker-1-thread-2] Connection channel unregistered: 172.17.223.67:12201 2021-09-27 23:57:49,332 INFO [ConnectionEventHandler#299] [bolt-netty-client-worker-1-thread-3] Connection channel registered: UNKNOWN-ADDR 2021-09-27 23:57:49,333 INFO [ConnectionEventHandler#80] [bolt-netty-client-worker-1-thread-3] Try connect to 172.17.223.67:12201 2021-09-27 23:57:49,334 INFO [ConnectionEventHandler#297] [bolt-netty-client-worker-1-thread-3] Connection channel active: 172.17.223.67:12201 2021-09-27 23:57:49,334 INFO [ConnectionEventHandler#175] [bolt-netty-client-worker-1-thread-3] trigger user event, local[172.17.198.144:59072], remote[172.17.223.67:12201], event: CONNECT 2021-09-27 23:58:11,553 WARN [ConnectionEventHandler#202] [bolt-netty-client-worker-1-thread-3] ExceptionCaught in connection: local[172.17.198.144:59072], remote[172.17.223.67:12201], close the connection! Cause[NativeIoException:readAddress(..) failed: l??????] 2021-09-27 23:58:11,553 INFO [ConnectionEventHandler#297] [bolt-netty-client-worker-1-thread-3] Connection closed: 172.17.223.67:12201 2021-09-27 23:58:11,553 INFO [ConnectionEventHandler#297] [bolt-netty-client-worker-1-thread-3] Connection closed: 172.17.223.67:12201 2021-09-27 23:58:11,553 INFO [ConnectionEventHandler#297] [bolt-netty-client-worker-1-thread-3] Connection channel inactive: 172.17.223.67:12201 2021-09-27 23:58:11,553 INFO [ConnectionEventHandler#175] [bolt-netty-client-worker-1-thread-3] trigger user event, local[172.17.198.144:59072], remote[172.17.223.67:12201], event: CLOSE 2021-09-27 23:58:11,553 INFO [ConnectionEventHandler#297] [bolt-netty-client-worker-1-thread-3] Connection channel unregistered: 172.17.223.67:12201 2021-09-27 23:58:19,338 INFO [ConnectionEventHandler#299] [bolt-netty-client-worker-1-thread-1] Connection channel registered: UNKNOWN-ADDR 2021-09-27 23:58:19,338 INFO [ConnectionEventHandler#80] [bolt-netty-client-worker-1-thread-1] Try connect to 172.17.223.67:12201 2021-09-27 23:58:19,340 INFO [ConnectionEventHandler#297] [bolt-netty-client-worker-1-thread-1] Connection channel active: 172.17.223.67:12201 2021-09-27 23:58:19,340 INFO [ConnectionEventHandler#175] [bolt-netty-client-worker-1-thread-1] trigger user event, local[172.17.198.144:59134], remote[172.17.223.67:12201], event: CONNECT

leizhiyuan commented 2 years ago

logs/rpc目录下又个rpc-client.log 日志我记得，这里的日志也发一下？

rabbit1013 commented 2 years ago

//这个是6点过得bolt得connection-event日志

2021-09-27 18:13:33,395 INFO [ConnectionEventHandler#297] [bolt-netty-client-worker-1-thread-1] Connection channel active: 172.17.223.67:12201 2021-09-27 18:13:33,395 INFO [ConnectionEventHandler#175] [bolt-netty-client-worker-1-thread-1] trigger user event, local[172.17.198.144:57052], remote[172.17.223.67:12201], event: CONNECT 2021-09-27 18:13:35,709 INFO [ConnectionEventHandler#299] [bolt-netty-client-worker-1-thread-2] Connection channel registered: UNKNOWN-ADDR 2021-09-27 18:13:35,710 INFO [ConnectionEventHandler#80] [bolt-netty-client-worker-1-thread-2] Try connect to 172.17.223.67:12208 2021-09-27 18:13:35,711 INFO [ConnectionEventHandler#297] [bolt-netty-client-worker-1-thread-2] Connection channel active: 172.17.223.67:12208 2021-09-27 18:13:35,712 INFO [ConnectionEventHandler#175] [bolt-netty-client-worker-1-thread-2] trigger user event, local[172.17.198.144:33670], remote[172.17.223.67:12208], event: CONNECT 2021-09-27 18:14:27,437 INFO [ConnectionEventHandler#299] [bolt-netty-client-worker-1-thread-1] Connection channel registered: UNKNOWN-ADDR 2021-09-27 18:14:27,438 INFO [ConnectionEventHandler#80] [bolt-netty-client-worker-1-thread-1] Try connect to 172.17.223.67:12210 2021-09-27 18:14:27,442 INFO [ConnectionEventHandler#297] [bolt-netty-client-worker-1-thread-1] Connection channel active: 172.17.223.67:12210 2021-09-27 18:14:27,447 INFO [ConnectionEventHandler#175] [bolt-netty-client-worker-1-thread-1] trigger user event, local[172.17.198.144:44650], remote[172.17.223.67:12210], event: CONNECT 2021-09-27 18:14:27,623 INFO [ConnectionEventHandler#299] [bolt-netty-client-worker-1-thread-2] Connection channel registered: UNKNOWN-ADDR 2021-09-27 18:14:27,642 INFO [ConnectionEventHandler#80] [bolt-netty-client-worker-1-thread-2] Try connect to 172.17.23.131:12222 2021-09-27 18:14:27,652 INFO [ConnectionEventHandler#297] [bolt-netty-client-worker-1-thread-2] Connection channel active: 172.17.23.131:12222 2021-09-27 18:14:27,652 INFO [ConnectionEventHandler#175] [bolt-netty-client-worker-1-thread-2] trigger user event, local[172.17.198.144:36314], remote[172.17.23.131:12222], event: CONNECT 2021-09-27 18:14:27,836 INFO [ConnectionEventHandler#299] [bolt-netty-client-worker-1-thread-3] Connection channel registered: UNKNOWN-ADDR 2021-09-27 18:14:27,837 INFO [ConnectionEventHandler#80] [bolt-netty-client-worker-1-thread-3] Try connect to 172.17.23.131:12211 2021-09-27 18:14:27,839 INFO [ConnectionEventHandler#297] [bolt-netty-client-worker-1-thread-3] Connection channel active: 172.17.23.131:12211 2021-09-27 18:14:27,839 INFO [ConnectionEventHandler#175] [bolt-netty-client-worker-1-thread-3] trigger user event, local[172.17.198.144:46692], remote[172.17.23.131:12211], event: CONNECT 2021-09-27 18:14:29,119 INFO [ConnectionEventHandler#299] [bolt-netty-client-worker-1-thread-1] Connection channel registered: UNKNOWN-ADDR 2021-09-27 18:14:29,119 INFO [ConnectionEventHandler#80] [bolt-netty-client-worker-1-thread-1] Try connect to 172.17.223.67:12201 2021-09-27 18:14:29,121 INFO [ConnectionEventHandler#297] [bolt-netty-client-worker-1-thread-1] Connection channel active: 172.17.223.67:12201 2021-09-27 18:14:29,123 INFO [ConnectionEventHandler#175] [bolt-netty-client-worker-1-thread-1] trigger user event, local[172.17.198.144:57162], remote[172.17.223.67:12201], event: CONNECT 2021-09-27 18:14:29,344 INFO [ConnectionEventHandler#299] [bolt-netty-client-worker-1-thread-2] Connection channel registered: UNKNOWN-ADDR 2021-09-27 18:14:29,344 INFO [ConnectionEventHandler#80] [bolt-netty-client-worker-1-thread-2] Try connect to 172.17.223.67:12208 2021-09-27 18:14:29,346 INFO [ConnectionEventHandler#297] [bolt-netty-client-worker-1-thread-2] Connection channel active: 172.17.223.67:12208 2021-09-27 18:14:29,354 INFO [ConnectionEventHandler#175] [bolt-netty-client-worker-1-thread-2] trigger user event, local[172.17.198.144:33770], remote[172.17.223.67:12208], event: CONNECT 2021-09-27 18:14:29,651 INFO [ConnectionEventHandler#299] [bolt-netty-client-worker-1-thread-3] Connection channel registered: UNKNOWN-ADDR 2021-09-27 18:14:29,651 INFO [ConnectionEventHandler#80] [bolt-netty-client-worker-1-thread-3] Try connect to 172.17.223.67:12223 2021-09-27 18:14:29,653 INFO [ConnectionEventHandler#297] [bolt-netty-client-worker-1-thread-3] Connection channel active: 172.17.223.67:12223

//这个是同一时间得rpc下面rpc-client得日志 2021-09-27 18:14:29,386 INFO SOFA-CLI-CONN-com.improve.edu.rpc.cms.CourseRpcService-40-T1 com.alipay.sofa.rpc.client.AllConnectConnectionHolder - Connect to com.improve.edu.rpc.cms.CourseRpcService provider:bolt://172.17.223.67:12208?accepts=100000&appName=imp-cms-server&delay=-1&dynamic=true&id=rpc-cfg-111&interface=com.improve.edu.rpc.cms.CourseRpcService&language=java&pid=8129&protocol=bolt&rpcVer=50709&serialization=hessian2&startTime=1632737616107&timeout=0&uniqueId=&weight=100 success ! The connection is 172.17.223.67:12208 <-> 172.17.198.144:33770 2021-09-27 18:14:29,431 INFO main com.alipay.sofa.rpc.client.AllConnectConnectionHolder - Add provider of com.improve.edu.rpc.cms.CourseUnitRpcService, size is : 1 2021-09-27 18:14:29,431 INFO SOFA-CLI-CONN-com.improve.edu.rpc.cms.CourseUnitRpcService-42-T1 com.alipay.sofa.rpc.client.AllConnectConnectionHolder - Connect to com.improve.edu.rpc.cms.CourseUnitRpcService provider:bolt://172.17.223.67:12208?accepts=100000&appName=imp-cms-server&delay=-1&dynamic=true&id=rpc-cfg-99&interface=com.improve.edu.rpc.cms.CourseUnitRpcService&language=java&pid=8129&protocol=bolt&rpcVer=50709&serialization=hessian2&startTime=1632737615916&timeout=0&uniqueId=&weight=100 success ! The connection is 172.17.223.67:12208 <-> 172.17.198.144:33770 2021-09-27 18:14:29,464 INFO main com.alipay.sofa.rpc.client.AllConnectConnectionHolder - Add provider of com.improve.edu.rpc.cms.CourseLevelRpcService, size is : 1 2021-09-27 18:14:29,464 INFO SOFA-CLI-CONN-com.improve.edu.rpc.cms.CourseLevelRpcService-44-T1 com.alipay.sofa.rpc.client.AllConnectConnectionHolder - Connect to com.improve.edu.rpc.cms.CourseLevelRpcService provider:bolt://172.17.223.67:12208?accepts=100000&appName=imp-cms-server&delay=-1&dynamic=true&id=rpc-cfg-108&interface=com.improve.edu.rpc.cms.CourseLevelRpcService&language=java&pid=8129&protocol=bolt&rpcVer=50709&serialization=hessian2&startTime=1632737615939&timeout=0&uniqueId=&weight=100 success ! The connection is 172.17.223.67:12208 <-> 172.17.198.144:33770 2021-09-27 18:14:29,507 INFO main com.alipay.sofa.rpc.client.AllConnectConnectionHolder - Add provider of com.improve.edu.rpc.user.EmployeeRpcService, size is : 1 2021-09-27 18:14:29,509 INFO SOFA-CLI-CONN-com.improve.edu.rpc.user.EmployeeRpcService-46-T1 com.alipay.sofa.rpc.client.AllConnectConnectionHolder - Connect to com.improve.edu.rpc.user.EmployeeRpcService provider:bolt://172.17.23.131:12222?accepts=100000&appName=sofa-user-server&delay=-1&dynamic=true&id=rpc-cfg-21&interface=com.improve.edu.rpc.user.EmployeeRpcService&language=java&pid=19758&protocol=bolt&rpcVer=50709&serialization=hessian2&startTime=1632737593660&timeout=0&uniqueId=&weight=100 success ! The connection is 172.17.23.131:12222 <-> 172.17.198.144:36314 2021-09-27 18:14:29,521 INFO main com.alipay.sofa.rpc.client.AllConnectConnectionHolder - Add provider of com.improve.edu.rpc.user.AccountRpcService, size is : 1 2021-09-27 18:14:29,521 INFO SOFA-CLI-CONN-com.improve.edu.rpc.user.AccountRpcService-48-T1 com.alipay.sofa.rpc.client.AllConnectConnectionHolder - Connect to com.improve.edu.rpc.user.AccountRpcService provider:bolt://172.17.23.131:12222?accepts=100000&appName=sofa-user-server&delay=-1&dynamic=true&id=rpc-cfg-22&interface=com.improve.edu.rpc.user.AccountRpcService&language=java&pid=19758&protocol=bolt&rpcVer=50709&serialization=hessian2&startTime=1632737593571&timeout=0&uniqueId=&weight=100 success ! The connection is 172.17.23.131:12222 <-> 172.17.198.144:36314 2021-09-27 18:14:29,638 INFO main com.alipay.sofa.rpc.client.AllConnectConnectionHolder - Add provider of com.improve.edu.rpc.trd.WxRpcService, size is : 1 2021-09-27 18:14:29,655 INFO SOFA-CLI-CONN-com.improve.edu.rpc.trd.WxRpcService-50-T1 com.alipay.sofa.rpc.client.AllConnectConnectionHolder - Connect to com.improve.edu.rpc.trd.WxRpcService provider:bolt://172.17.223.67:12223?accepts=100000&appName=imp-integration-service&delay=-1&dynamic=true&id=rpc-cfg-17&interface=com.improve.edu.rpc.trd.WxRpcService&language=java&pid=8175&protocol=bolt&rpcVer=50709&serialization=hessian2&startTime=1632737603068&timeout=0&uniqueId=&weight=100 success ! The connection is 172.17.223.67:12223 <-> 172.17.198.144:51054 2021-09-27 18:14:29,699 INFO main com.alipay.sofa.rpc.client.AllConnectConnectionHolder - Add provider of com.improve.edu.rpc.stum.StudentAccountRpcService, size is : 1 2021-09-27 18:14:29,699 INFO SOFA-CLI-CONN-com.improve.edu.rpc.stum.StudentAccountRpcService-52-T1 com.alipay.sofa.rpc.client.AllConnectConnectionHolder - Connect to com.improve.edu.rpc.stum.StudentAccountRpcService provider:bolt://172.17.223.67:12210?accepts=100000&appName=imp-stum-server&delay=-1&dynamic=true&id=rpc-cfg-55&interface=com.improve.edu.rpc.stum.StudentAccountRpcService&language=java&pid=8232&protocol=bolt&rpcVer=50709&serialization=hessian2&startTime=1632737605852&timeout=0&uniqueId=&weight=100 success ! The connection is 172.17.223.67:12210 <-> 172.17.198.144:44650 2021-09-27 18:14:29,722 INFO main com.alipay.sofa.rpc.client.AllConnectConnectionHolder - Add provider of com.improve.edu.rpc.trd.SmsRpcService, size is : 1 2021-09-27 18:14:29,722 INFO SOFA-CLI-CONN-com.improve.edu.rpc.trd.SmsRpcService-54-T1 com.alipay.sofa.rpc.client.AllConnectConnectionHolder - Connect to com.improve.edu.rpc.trd.SmsRpcService provider:bolt://172.17.223.67:12223?accepts=100000&appName=imp-integration-service&delay=-1&dynamic=true&id=rpc-cfg-14&interface=com.improve.edu.rpc.trd.SmsRpcService&language=java&pid=8175&protocol=bolt&rpcVer=50709&serialization=hessian2&startTime=1632737603137&timeout=4000&uniqueId=&weight=100 success ! The connection is 172.17.223.67:12223 <-> 172.17.198.144:51054 2021-09-27 18:14:29,736 INFO main com.alipay.sofa.rpc.client.AllConnectConnectionHolder - Add provider of com.improve.edu.rpc.tms.StudentMiniRpcService, size is : 1 2021-09-27 18:14:29,737 INFO SOFA-CLI-CONN-com.improve.edu.rpc.tms.StudentMiniRpcService-56-T1 com.alipay.sofa.rpc.client.AllConnectConnectionHolder - Connect to com.improve.edu.rpc.tms.StudentMiniRpcService provider:bolt://172.17.223.67:12201?accepts=100000&appName=imp-tms-server&delay=-1&dynamic=true&id=rpc-cfg-60&interface=com.improve.edu.rpc.tms.StudentMiniRpcService&language=java&pid=8264&protocol=bolt&rpcVer=50709&serialization=hessian2&startTime=1632737612574&timeout=0&uniqueId=&weight=100 success ! The connection is 172.17.223.67:12201 <-> 172.17.198.144:57162 2021-09-27 18:14:29,756 INFO main com.alipay.sofa.rpc.client.AllConnectConnectionHolder - Add provider of com.improve.edu.rpc.tms.TmsRpcService, size is : 1 2021-09-27 18:14:29,757 INFO SOFA-CLI-CONN-com.improve.edu.rpc.tms.TmsRpcService-58-T1 com.alipay.sofa.rpc.client.AllConnectConnectionHolder - Connect to com.improve.edu.rpc.tms.TmsRpcService provider:bolt://172.17.223.67:12201?accepts=100000&appName=imp-tms-server&delay=-1&dynamic=true&id=rpc-cfg-53&interface=com.improve.edu.rpc.tms.TmsRpcService&language=java&pid=8264&protocol=bolt&rpcVer=50709&serialization=hessian2&startTime=1632737612381&timeout=0&uniqueId=&weight=100 success ! The connection is 172.17.223.67:12201 <-> 172.17.198.144:57162 2021-09-27 18:14:29,765 INFO main com.alipay.sofa.rpc.client.AllConnectConnectionHolder - Add provider of com.improve.edu.rpc.tms.AppStudentInfoRpcService, size is : 1 2021-09-27 18:14:29,765 INFO SOFA-CLI-CONN-com.improve.edu.rpc.tms.AppStudentInfoRpcService-60-T1 com.alipay.sofa.rpc.client.AllConnectConnectionHolder - Connect to com.improve.edu.rpc.tms.AppStudentInfoRpcService provider:bolt://172.17.223.67:12201?accepts=100000&appName=imp-tms-server&delay=-1&dynamic=true&id=rpc-cfg-63&interface=com.improve.edu.rpc.tms.AppStudentInfoRpcService&language=java&pid=8264&protocol=bolt&rpcVer=50709&serialization=hessian2&startTime=1632737612456&timeout=0&uniqueId=&weight=100 success ! The connection is 172.17.223.67:12201 <-> 172.17.198.144:57162 2021-09-27 18:14:29,831 INFO main com.alipay.sofa.rpc.client.AllConnectConnectionHolder - Add provider of com.improve.edu.rpc.tms.StudentCourseRpcService, size is : 1 2021-09-27 18:14:29,832 INFO SOFA-CLI-CONN-com.improve.edu.rpc.tms.StudentCourseRpcService-62-T1 com.alipay.sofa.rpc.client.AllConnectConnectionHolder - Connect to com.improve.edu.rpc.tms.StudentCourseRpcService provider:bolt://172.17.223.67:12201?accepts=100000&appName=imp-tms-server&delay=-1&dynamic=true&id=rpc-cfg-62&interface=com.improve.edu.rpc.tms.StudentCourseRpcService&language=java&pid=8264&protocol=bolt&rpcVer=50709&serialization=hessian2&startTime=1632737612491&timeout=0&uniqueId=&weight=100 success ! The connection is 172.17.223.67:12201 <-> 172.17.198.144:57162 2021-09-27 18:14:29,839 INFO main com.alipay.sofa.rpc.client.AllConnectConnectionHolder - Add provider of com.improve.edu.rpc.stum.StudentManagementRpcService, size is : 1 2021-09-27 18:14:29,857 INFO SOFA-CLI-CONN-com.improve.edu.rpc.stum.StudentManagementRpcService-64-T1 com.alipay.sofa.rpc.client.AllConnectConnectionHolder - Connect to com.improve.edu.rpc.stum.StudentManagementRpcService provider:bolt://172.17.223.67:12210?accepts=100000&appName=imp-stum-server&delay=-1&dynamic=true&id=rpc-cfg-59&interface=com.improve.edu.rpc.stum.StudentManagementRpcService&language=java&pid=8232&protocol=bolt&rpcVer=50709&serialization=hessian2&startTime=1632737605796&timeout=6000&uniqueId=&weight=100 success ! The connection is 172.17.223.67:12210 <-> 172.17.198.144:44650 2021-09-27 18:14:29,863 INFO main com.alipay.sofa.rpc.client.AllConnectConnectionHolder - Add provider of com.improve.edu.rpc.stum.AssessmentRecordMiniRpcService, size is : 1 2021-09-27 18:14:29,864 INFO SOFA-CLI-CONN-com.improve.edu.rpc.stum.AssessmentRecordMiniRpcService-66-T1 com.alipay.sofa.rpc.client.AllConnectConnectionHolder - Connect to com.improve.edu.rpc.stum.AssessmentRecordMiniRpcService provider:bolt://172.17.223.67:12210?accepts=100000&appName=imp-stum-server&delay=-1&dynamic=true&id=rpc-cfg-46&interface=com.improve.edu.rpc.stum.AssessmentRecordMiniRpcService&language=java&pid=8232&protocol=bolt&rpcVer=50709&serialization=hessian2&startTime=1632737605805&timeout=0&uniqueId=&weight=100 success ! The connection is 172.17.223.67:12210 <-> 172.17.198.144:44650 2021-09-27 18:14:29,887 INFO main com.alipay.sofa.rpc.client.AllConnectConnectionHolder - Add provider of com.improve.edu.rpc.tms.TeacherWorkRpcService, size is : 1 2021-09-27 18:14:29,889 INFO SOFA-CLI-CONN-com.improve.edu.rpc.tms.TeacherWorkRpcService-68-T1 com.alipay.sofa.rpc.client.AllConnectConnectionHolder - Connect to com.improve.edu.rpc.tms.TeacherWorkRpcService provider:bolt://172.17.223.67:12201?accepts=100000&appName=imp-tms-server&delay=-1&dynamic=true&id=rpc-cfg-51&interface=com.improve.edu.rpc.tms.TeacherWorkRpcService&language=java&pid=8264&protocol=bolt&rpcVer=50709&serialization=hessian2&startTime=1632737612472&timeout=0&uniqueId=&weight=100 success ! The connection is 172.17.223.67:12201 <-> 172.17.198.144:57162 2021-09-27 18:14:29,926 INFO main com.alipay.sofa.rpc.client.AllConnectConnectionHolder - Add provider of com.improve.edu.rpc.cms.ReadingRpcService, size is : 1 2021-09-27 18:14:29,928 INFO SOFA-CLI-CONN-com.improve.edu.rpc.cms.ReadingRpcService-70-T1 com.alipay.sofa.rpc.client.AllConnectConnectionHolder - Connect to com.improve.edu.rpc.cms.ReadingRpcService provider:bolt://172.17.223.67:12208?accepts=100000&appName=imp-cms-server&delay=-1&dynamic=true&id=rpc-cfg-113&interface=com.improve.edu.rpc.cms.ReadingRpcService&language=java&pid=8129&protocol=bolt&rpcVer=50709&serialization=hessian2&startTime=1632737616411&timeout=0&uniqueId=&weight=100 success ! The connection is 172.17.223.67:12208 <-> 172.17.198.144:33770 2021-09-27 18:14:29,983 INFO main com.alipay.sofa.rpc.client.AllConnectConnectionHolder - Add provider of com.improve.edu.rpc.cms.ReadingCategoryRpcService, size is : 1 2021-09-27 18:14:29,985 INFO SOFA-CLI-CONN-com.improve.edu.rpc.cms.ReadingCategoryRpcService-72-T1 com.alipay.sofa.rpc.client.AllConnectConnectionHolder - Connect to com.improve.edu.rpc.cms.ReadingCategoryRpcService provider:bolt://172.17.223.67:12208?accepts=100000&appName=imp-cms-server&delay=-1&dynamic=true&id=rpc-cfg-94&interface=com.improve.edu.rpc.cms.ReadingCategoryRpcService&language=java&pid=8129&protocol=bolt&rpcVer=50709&serialization=hessian2&startTime=1632737616013&timeout=0&uniqueId=&weight=100 success ! The connection is 172.17.223.67:12208 <-> 172.17.198.144:33770 2021-09-27 18:14:30,019 INFO main com.alipay.sofa.rpc.client.AllConnectConnectionHolder - Add provider of com.improve.edu.rpc.cms.TmsClassLevelRpcService, size is : 1 2021-09-27 18:14:30,020 INFO SOFA-CLI-CONN-com.improve.edu.rpc.cms.TmsClassLevelRpcService-74-T1 com.alipay.sofa.rpc.client.AllConnectConnectionHolder - Connect to com.improve.edu.rpc.cms.TmsClassLevelRpcService provider:bolt://172.17.223.67:12208?accepts=100000&appName=imp-cms-server&delay=-1&dynamic=true&id=rpc-cfg-42&interface=com.improve.edu.rpc.cms.TmsClassLevelRpcService&language=java&pid=8129&protocol=bolt&rpcVer=50709&serialization=hessian2&startTime=1632737616367&timeout=0&uniqueId=&weight=100 success ! The connection is 172.17.223.67:12208 <-> 172.17.198.144:33770 2021-09-27 18:14:30,031 INFO main com.alipay.sofa.rpc.client.AllConnectConnectionHolder - Add provider of com.improve.edu.rpc.tms.TeacherWorkRpcService, size is : 1 2021-09-27 18:14:30,033 INFO SOFA-CLI-CONN-com.improve.edu.rpc.tms.TeacherWorkRpcService-76-T1 com.alipay.sofa.rpc.client.AllConnectConnectionHolder - Connect to com.improve.edu.rpc.tms.TeacherWorkRpcService provider:bolt://172.17.223.67:12201?accepts=100000&appName=imp-tms-server&delay=-1&dynamic=true&id=rpc-cfg-51&interface=com.improve.edu.rpc.tms.TeacherWorkRpcService&language=java&pid=8264&protocol=bolt&rpcVer=50709&serialization=hessian2&startTime=1632737612472&timeout=0&uniqueId=&weight=100 success ! The connection is 172.17.223.67:12201 <-> 172.17.198.144:57162 2021-09-27 18:14:30,034 INFO main com.alipay.sofa.rpc.client.AllConnectConnectionHolder - Add provider of com.improve.edu.rpc.cms.CourseLevelRpcService, size is : 1 2021-09-27 18:14:30,036 INFO SOFA-CLI-CONN-com.improve.edu.rpc.cms.CourseLevelRpcService-78-T1 com.alipay.sofa.rpc.client.AllConnectConnectionHolder - Connect to com.improve.edu.rpc.cms.CourseLevelRpcService provider:bolt://172.17.223.67:12208?accepts=100000&appName=imp-cms-server&delay=-1&dynamic=true&id=rpc-cfg-108&interface=com.improve.edu.rpc.cms.CourseLevelRpcService&language=java&pid=8129&protocol=bolt&rpcVer=50709&serialization=hessian2&startTime=1632737615939&timeout=0&uniqueId=&weight=100 success ! The connection is 172.17.223.67:12208 <-> 172.17.198.144:33770 2021-09-27 18:14:30,042 INFO main com.alipay.sofa.rpc.client.AllConnectConnectionHolder - Add provider of com.improve.edu.rpc.cms.LessonHomeworkRpcService, size is : 1 2021-09-27 18:14:30,043 INFO SOFA-CLI-CONN-com.improve.edu.rpc.cms.LessonHomeworkRpcService-80-T1 com.alipay.sofa.rpc.client.AllConnectConnectionHolder - Connect to com.improve.edu.rpc.cms.LessonHomeworkRpcService provider:bolt://172.17.223.67:12208?accepts=100000&appName=imp-cms-server&delay=-1&dynamic=true&id=rpc-cfg-58&interface=com.improve.edu.rpc.cms.LessonHomeworkRpcService&language=java&pid=8129&protocol=bolt&rpcVer=50709&serialization=hessian2&startTime=1632737616136&timeout=0&uniqueId=&weight=100 success ! The connection is 172.17.223.67:12208 <-> 172.17.198.144:33770 2021-09-27 18:14:30,053 INFO main com.alipay.sofa.rpc.client.AllConnectConnectionHolder - Add provider of com.improve.edu.rpc.cms.CourseRpcService, size is : 1 2021-09-27 18:14:30,054 INFO SOFA-CLI-CONN-com.improve.edu.rpc.cms.CourseRpcService-82-T1 com.alipay.sofa.rpc.client.AllConnectConnectionHolder - Connect to com.improve.edu.rpc.cms.CourseRpcService provider:bolt://172.17.223.67:12208?accepts=100000&appName=imp-cms-server&delay=-1&dynamic=true&id=rpc-cfg-111&interface=com.improve.edu.rpc.cms.CourseRpcService&language=java&pid=8129&protocol=bolt&rpcVer=50709&serialization=hessian2&startTime=1632737616107&timeout=0&uniqueId=&weight=100 success ! The connection is 172.17.223.67:12208 <-> 172.17.198.144:33770 2021-09-27 18:14:30,059 INFO main com.alipay.sofa.rpc.client.AllConnectConnectionHolder - Add provider of com.improve.edu.rpc.cms.MiniCourseRpcService, size is : 1 2021-09-27 18:14:30,060 INFO SOFA-CLI-CONN-com.improve.edu.rpc.cms.MiniCourseRpcService-84-T1 com.alipay.sofa.rpc.client.AllConnectConnectionHolder - Connect to com.improve.edu.rpc.cms.MiniCourseRpcService provider:bolt://172.17.223.67:12208?accepts=100000&appName=imp-cms-server&delay=-1&dynamic=true&id=rpc-cfg-45&interface=com.improve.edu.rpc.cms.MiniCourseRpcService&language=java&pid=8129&protocol=bolt&rpcVer=50709&serialization=hessian2&startTime=1632737616336&timeout=0&uniqueId=&weight=100 success ! The connection is 172.17.223.67:12208 <-> 172.17.198.144:33770 2021-09-27 18:14:30,078 INFO main com.alipay.sofa.rpc.client.AllConnectConnectionHolder - Add provider of com.improve.edu.rpc.tms.StudentInfoRpcService, size is : 1 2021-09-27 18:14:30,078 INFO SOFA-CLI-CONN-com.improve.edu.rpc.tms.StudentInfoRpcService-86-T1 com.alipay.sofa.rpc.client.AllConnectConnectionHolder - Connect to com.improve.edu.rpc.tms.StudentInfoRpcService provider:bolt://172.17.223.67:12201?accepts=100000&appName=imp-tms-server&delay=-1&dynamic=true&id=rpc-cfg-50&interface=com.improve.edu.rpc.tms.StudentInfoRpcService&language=java&pid=8264&protocol=bolt&rpcVer=50709&serialization=hessian2&startTime=1632737612411&timeout=0&uniqueId=&weight=100 success ! The connection is 172.17.223.67:12201 <-> 172.17.198.144:57162 2021-09-27 18:14:30,090 INFO main com.alipay.sofa.rpc.client.AllConnectConnectionHolder - Add provider of com.improve.edu.rpc.cms.MiniCourseLessonRpcService, size is : 1 2021-09-27 18:14:30,090 INFO SOFA-CLI-CONN-com.improve.edu.rpc.cms.MiniCourseLessonRpcService-88-T1 com.alipay.sofa.rpc.client.AllConnectConnectionHolder - Connect to com.improve.edu.rpc.cms.MiniCourseLessonRpcService provider:bolt://172.17.223.67:12208?accepts=100000&appName=imp-cms-server&delay=-1&dynamic=true&id=rpc-cfg-76&interface=com.improve.edu.rpc.cms.MiniCourseLessonRpcService&language=java&pid=8129&protocol=bolt&rpcVer=50709&serialization=hessian2&startTime=1632737616169&timeout=0&uniqueId=&weight=100 success ! The connection is 172.17.223.67:12208 <-> 172.17.198.144:33770 2021-09-27 18:14:30,138 INFO main com.alipay.sofa.rpc.client.AllConnectConnectionHolder - Add provider of com.improve.edu.rpc.cms.MiniCourseLevelRpcService, size is : 1 2021-09-27 18:14:30,138 INFO SOFA-CLI-CONN-com.improve.edu.rpc.cms.MiniCourseLevelRpcService-90-T1 com.alipay.sofa.rpc.client.AllConnectConnectionHolder - Connect to com.improve.edu.rpc.cms.MiniCourseLevelRpcService provider:bolt://172.17.223.67:12208?accepts=100000&appName=imp-cms-server&delay=-1&dynamic=true&id=rpc-cfg-54&interface=com.improve.edu.rpc.cms.MiniCourseLevelRpcService&language=java&pid=8129&protocol=bolt&rpcVer=50709&serialization=hessian2&startTime=1632737616114&timeout=0&uniqueId=&weight=100 success ! The connection is 172.17.223.67:12208 <-> 172.17.198.144:33770 2021-09-27 18:14:30,151 INFO main com.alipay.sofa.rpc.client.AllConnectConnectionHolder - Add provider of com.improve.edu.rpc.cms.MiniLessonAdditionalRpcService, size is : 1 2021-09-27 18:14:30,152 INFO SOFA-CLI-CONN-com.improve.edu.rpc.cms.MiniLessonAdditionalRpcService-92-T1 com.alipay.sofa.rpc.client.AllConnectConnectionHolder - Connect to com.improve.edu.rpc.cms.MiniLessonAdditionalRpcService provider:bolt://172.17.223.67:12208?accepts=100000&appName=imp-cms-server&delay=-1&dynamic=true&id=rpc-cfg-48&interface=com.improve.edu.rpc.cms.MiniLessonAdditionalRpcService&language=java&pid=8129&protocol=bolt&rpcVer=50709&serialization=hessian2&startTime=1632737616396&timeout=0&uniqueId=&weight=100 success ! The connection is 172.17.223.67:12208 <-> 172.17.198.144:33770 2021-09-27 18:14:30,173 INFO main com.alipay.sofa.rpc.client.AllConnectConnectionHolder - Add provider of com.improve.edu.rpc.edms.TeacherWorkHomeworkRpcService, size is : 1 2021-09-27 18:14:30,174 INFO SOFA-CLI-CONN-com.improve.edu.rpc.edms.TeacherWorkHomeworkRpcService-94-T1 com.alipay.sofa.rpc.client.AllConnectConnectionHolder - Connect to com.improve.edu.rpc.edms.TeacherWorkHomeworkRpcService provider:bolt://172.17.223.67:12201?accepts=100000&appName=imp-tms-server&delay=-1&dynamic=true&id=rpc-cfg-45&interface=com.improve.edu.rpc.edms.TeacherWorkHomeworkRpcService&language=java&pid=8264&protocol=bolt&rpcVer=50709&serialization=hessian2&startTime=1632737612507&timeout=0&uniqueId=&weight=100 success ! The connection is 172.17.223.67:12201 <-> 172.17.198.144:57162 2021-09-27 18:14:30,193 INFO main com.alipay.sofa.rpc.client.AllConnectConnectionHolder - Add provider of com.improve.edu.rpc.stum.HomeworkRecordRpcService, size is : 1 2021-09-27 18:14:30,193 INFO SOFA-CLI-CONN-com.improve.edu.rpc.stum.HomeworkRecordRpcService-96-T1 com.alipay.sofa.rpc.client.AllConnectConnectionHolder - Connect to com.improve.edu.rpc.stum.HomeworkRecordRpcService provider:bolt://172.17.223.67:12210?accepts=100000&appName=imp-stum-server&delay=-1&dynamic=true&id=rpc-cfg-61&interface=com.improve.edu.rpc.stum.HomeworkRecordRpcService&language=java&pid=8232&protocol=bolt&rpcVer=50709&serialization=hessian2&startTime=1632737605813&timeout=0&uniqueId=&weight=100 success ! The connection is 172.17.223.67:12210 <-> 172.17.198.144:44650 2021-09-27 18:14:30,198 INFO main com.alipay.sofa.rpc.client.AllConnectConnectionHolder - Add provider of com.improve.edu.rpc.cms.CourseLessonRpcService, size is : 1 2021-09-27 18:14:30,198 INFO SOFA-CLI-CONN-com.improve.edu.rpc.cms.CourseLessonRpcService-98-T1 com.alipay.sofa.rpc.client.AllConnectConnectionHolder - Connect to com.improve.edu.rpc.cms.CourseLessonRpcService provider:bolt://172.17.223.67:12208?accepts=100000&appName=imp-cms-server&delay=-1&dynamic=true&id=rpc-cfg-47&interface=com.improve.edu.rpc.cms.CourseLessonRpcService&language=java&pid=8129&protocol=bolt&rpcVer=50709&serialization=hessian2&startTime=1632737615841&timeout=0&uniqueId=&weight=100 success ! The connection is 172.17.223.67:12208 <-> 172.17.198.144:33770 2021-09-27 18:14:30,218 INFO main com.alipay.sofa.rpc.client.AllConnectConnectionHolder - Add provider of com.improve.edu.rpc.stum.AppSpeakRecordRpcService, size is : 1 2021-09-27 18:14:30,219 INFO SOFA-CLI-CONN-com.improve.edu.rpc.stum.AppSpeakRecordRpcService-100-T1 com.alipay.sofa.rpc.client.AllConnectConnectionHolder - Connect to com.improve.edu.rpc.stum.AppSpeakRecordRpcService provider:bolt://172.17.223.67:12210?accepts=100000&appName=imp-stum-server&delay=-1&dynamic=true&id=rpc-cfg-48&interface=com.improve.edu.rpc.stum.AppSpeakRecordRpcService&language=java&pid=8232&protocol=bolt&rpcVer=50709&serialization=hessian2&startTime=1632737605738&timeout=0&uniqueId=&weight=100 success ! The connection is 172.17.223.67:12210 <-> 172.17.198.144:44650 2021-09-27 18:14:30,270 INFO main com.alipay.sofa.rpc.client.AllConnectConnectionHolder - Add provider of com.improve.edu.rpc.sys.SysMessageRpcService, size is : 1 2021-09-27 18:14:30,285 INFO SOFA-CLI-CONN-com.improve.edu.rpc.sys.SysMessageRpcService-102-T1 com.alipay.sofa.rpc.client.AllConnectConnectionHolder - Connect to com.improve.edu.rpc.sys.SysMessageRpcService provider:bolt://172.17.223.67:12209?accepts=100000&appName=imp-common-msg&delay=-1&dynamic=true&id=rpc-cfg-8&interface=com.improve.edu.rpc.sys.SysMessageRpcService&language=java&pid=8208&protocol=bolt&rpcVer=50709&serialization=hessian2&startTime=1632737597447&timeout=0&uniqueId=&weight=100 success ! The connection is 172.17.223.67:12209 <-> 172.17.198.144:36746 2021-09-27 18:14:30,313 INFO main com.alipay.sofa.rpc.client.AllConnectConnectionHolder - Add provider of com.improve.edu.rpc.cms.PointReadRpcService, size is : 1 2021-09-27 18:14:30,314 INFO SOFA-CLI-CONN-com.improve.edu.rpc.cms.PointReadRpcService-104-T1 com.alipay.sofa.rpc.client.AllConnectConnectionHolder - Connect to com.improve.edu.rpc.cms.PointReadRpcService provider:bolt://172.17.223.67:12208?accepts=100000&appName=imp-cms-server&delay=-1&dynamic=true&id=rpc-cfg-50&interface=com.improve.edu.rpc.cms.PointReadRpcService&language=java&pid=8129&protocol=bolt&rpcVer=50709&serialization=hessian2&startTime=1632737615947&timeout=0&uniqueId=&weight=100 success ! The connection is 172.17.223.67:12208 <-> 172.17.198.144:33770 2021-09-27 18:14:30,329 INFO main com.alipay.sofa.rpc.client.AllConnectConnectionHolder - Add provider of com.improve.edu.rpc.cms.PointReadCategoryRpcService, size is : 1 2021-09-27 18:14:30,330 INFO SOFA-CLI-CONN-com.improve.edu.rpc.cms.PointReadCategoryRpcService-106-T1 com.alipay.sofa.rpc.client.AllConnectConnectionHolder - Connect to com.improve.edu.rpc.cms.PointReadCategoryRpcService provider:bolt://172.17.223.67:12208?accepts=100000&appName=imp-cms-server&delay=-1&dynamic=true&id=rpc-cfg-62&interface=com.improve.edu.rpc.cms.PointReadCategoryRpcService&language=java&pid=8129&protocol=bolt&rpcVer=50709&serialization=hessian2&startTime=1632737615794&timeout=0&uniqueId=&weight=100 success ! The connection is 172.17.223.67:12208 <-> 172.17.198.144:33770 2021-09-27 18:14:30,350 INFO main com.alipay.sofa.rpc.client.AllConnectConnectionHolder - Add provider of com.improve.edu.rpc.cms.PointReadCatalogRpcService, size is : 1 2021-09-27 18:14:30,351 INFO SOFA-CLI-CONN-com.improve.edu.rpc.cms.PointReadCatalogRpcService-108-T1 com.alipay.sofa.rpc.client.AllConnectConnectionHolder - Connect to com.improve.edu.rpc.cms.PointReadCatalogRpcService provider:bolt://172.17.223.67:12208?accepts=100000&appName=imp-cms-server&delay=-1&dynamic=true&id=rpc-cfg-95&interface=com.improve.edu.rpc.cms.PointReadCatalogRpcService&language=java&pid=8129&protocol=bolt&rpcVer=50709&serialization=hessian2&startTime=1632737616157&timeout=0&uniqueId=&weight=100 success ! The connection is 172.17.223.67:12208 <-> 172.17.198.144:33770 2021-09-27 18:14:30,362 INFO main com.alipay.sofa.rpc.client.AllConnectConnectionHolder - Add provider of com.improve.edu.rpc.cms.PointReadQuestionRpcService, size is : 1 2021-09-27 18:14:30,362 INFO SOFA-CLI-CONN-com.improve.edu.rpc.cms.PointReadQuestionRpcService-110-T1 com.alipay.sofa.rpc.client.AllConnectConnectionHolder - Connect to com.improve.edu.rpc.cms.PointReadQuestionRpcService provider:bolt://172.17.223.67:12208?accepts=100000&appName=imp-cms-server&delay=-1&dynamic=true&id=rpc-cfg-40&interface=com.improve.edu.rpc.cms.PointReadQuestionRpcService&language=java&pid=8129&protocol=bolt&rpcVer=50709&serialization=hessian2&startTime=1632737615785&timeout=0&uniqueId=&weight=100 success ! The connection is 172.17.223.67:12208 <-> 172.17.198.144:33770 2021-09-27 18:14:30,388 INFO main com.alipay.sofa.rpc.client.AllConnectConnectionHolder - Add provider of com.improve.edu.rpc.user.PermissonRpcService, size is : 1 2021-09-27 18:14:30,388 INFO SOFA-CLI-CONN-com.improve.edu.rpc.user.PermissonRpcService-112-T1 com.alipay.sofa.rpc.client.AllConnectConnectionHolder - Connect to com.improve.edu.rpc.user.PermissonRpcService provider:bolt://172.17.23.131:12222?accepts=100000&appName=sofa-user-server&delay=-1&dynamic=true&id=rpc-cfg-18&interface=com.improve.edu.rpc.user.PermissonRpcService&language=java&pid=19758&protocol=bolt&rpcVer=50709&serialization=hessian2&startTime=1632737593644&timeout=0&uniqueId=&weight=100 success ! The connection is 172.17.23.131:12222 <-> 172.17.198.144:36314 2021-09-27 18:14:30,400 INFO main com.alipay.sofa.rpc.client.AllConnectConnectionHolder - Add provider of com.improve.edu.rpc.sys.SysRpcService, size is : 1 2021-09-27 18:14:30,406 INFO SOFA-CLI-CONN-com.improve.edu.rpc.sys.SysRpcService-114-T1 com.alipay.sofa.rpc.client.AllConnectConnectionHolder - Connect to com.improve.edu.rpc.sys.SysRpcService provider:bolt://172.17.23.131:12213?accepts=100000&appName=imp-sys-server&delay=-1&dynamic=true&id=rpc-cfg-29&interface=com.improve.edu.rpc.sys.SysRpcService&language=java&pid=19780&protocol=bolt&rpcVer=50709&serialization=hessian2&startTime=1632737591579&timeout=0&uniqueId=&weight=100 success ! The connection is 172.17.23.131:12213 <-> 172.17.198.144:38864 2021-09-27 18:14:30,430 INFO main com.alipay.sofa.rpc.client.AllConnectConnectionHolder - Add provider of com.improve.edu.rpc.user.PermissonRpcService, size is : 1 2021-09-27 18:14:30,431 INFO SOFA-CLI-CONN-com.improve.edu.rpc.user.PermissonRpcService-116-T1 com.alipay.sofa.rpc.client.AllConnectConnectionHolder - Connect to com.improve.edu.rpc.user.PermissonRpcService provider:bolt://172.17.23.131:12222?accepts=100000&appName=sofa-user-server&delay=-1&dynamic=true&id=rpc-cfg-18&interface=com.improve.edu.rpc.user.PermissonRpcService&language=java&pid=19758&protocol=bolt&rpcVer=50709&serialization=hessian2&startTime=1632737593644&timeout=0&uniqueId=&weight=100 success ! The connection is 172.17.23.131:12222 <-> 172.17.198.144:36314 2021-09-27 18:14:31,979 INFO main com.alipay.sofa.rpc.client.aft.FaultToleranceConfigManager - Get a new resource, value[FaultToleranceConfig{timeWindow=10, leastCallCount=100, leastWindowCount=10, leastWindowExceptionRateMultiple=6.0, regulationEffective=false, weightDegradeRate=0.05, degradeEffective=false, degradeLeastWeight=1, weightRecoverRate=2.0, degradeMaxIpCount=2}]

dengbencheng commented 2 years ago

logs.zip 2021-10-03这一天的20点-21点服务有类似的异常报错信息，相关的bolt以及rpc日志文件已贴出。 请问这是什么问题造成的？

leizhiyuan commented 2 years ago

2021-10-03 20:43:09,848 WARN [RpcResponseResolver#131] [http-nio-8555-exec-9] Server deserialize request exception! the address is 172.17.23.131:12211, id=139379, serverSide=true

看到有大量的服务端反序列化异常，去这个机器上172.17.23.131看一下bolt和rpc下面的日志。里面应该有具体的反序列化错误。

rabbit1013 commented 2 years ago

我看到这里面有一个issue 写得是因为ConnectionClosedException 导致了服务器反序列化异常，重新启动就没有了 这个是因为反序列化异常导致得ConnectionClosedException这个异常得出现么？

leizhiyuan commented 2 years ago

得看这个反序列化异常的时候，是不是协议层面的， 如果是业务层面的，不会短链的。你看看服务端这个反序列化的具体错误是什么

dengbencheng commented 2 years ago

logs-10-03.zip 172.17.23.131服务器上10月3日的相关日志

leizhiyuan commented 2 years ago

文件下载不了？

dengbencheng commented 2 years ago

logs-10-03.zip 不好意思，昨天那个下载不了

leizhiyuan commented 2 years ago 
2021-10-05 08:21:20,999 WARN  [ConnectionEventHandler#202] [Rpc-netty-server-worker-6-thread-2] ExceptionCaught in connection: local[172.17.23.131:12211], remote[172.17.198.144:48848], close the connection! Cause[DecoderException:java.lang.IndexOutOfBoundsException: readerIndex(49717) + length(21106) exceeds writerIndex(65536): PooledUnsafeDirectByteBuf(ridx: 49717, widx: 65536, cap: 65536)]
2021-10-05 09:24:15,111 WARN  [ConnectionEventHandler#202] [Rpc-netty-server-worker-6-thread-4] ExceptionCaught in connection: local[172.17.23.131:12211], remote[172.17.198.144:50474], close the connection! Cause[DecoderException:com.alipay.remoting.exception.CodecException: Unknown protocol code: [ProtocolVersion{version=[79]}] while decode in ProtocolDecoder.]

2021-10-03 20:24:57,999 ERROR [RpcRequestProcessor#270] [SOFA-SEV-BOLT-BIZ-12211-14-T12] DeserializationException occurred when process in RpcRequestProcessor, id=138030, deserializeLevel=DESERIALIZE_ALL
com.alipay.remoting.exception.DeserializationException: Head of request is null or is not map
    at com.alipay.sofa.rpc.codec.bolt.SofaRpcSerialization.deserializeContent(SofaRpcSerialization.java:240)
    at com.alipay.remoting.rpc.protocol.RpcRequestCommand.deserializeContent(RpcRequestCommand.java:144)
    at com.alipay.remoting.rpc.RpcCommand.deserialize(RpcCommand.java:117)
    at com.alipay.remoting.rpc.RpcCommand.deserialize(RpcCommand.java:138)
    at com.alipay.remoting.rpc.protocol.RpcRequestProcessor.deserializeRequestCommand(RpcRequestProcessor.java:267)
    at com.alipay.remoting.rpc.protocol.RpcRequestProcessor.doProcess(RpcRequestProcessor.java:142)
    at com.alipay.remoting.rpc.protocol.RpcRequestProcessor$ProcessTask.run(RpcRequestProcessor.java:366)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:829)

原因看是这个原因了，但是这个问题从来没出现过，是不是你们做了什么操作，或者客户端和服务端的rpc版本有较大的差异，出现了协议层的问题。

不知道能不能提供一下你们使用的最简化的demo复现场景？

stale[bot] commented 2 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

it-linnan commented 9 months ago

最近遇到了类似问题，在这里分享下最后排查到的原因。

我们使用了 SOFARPC + SOFATracer，通过 SOFATracer 上报一些公共数据，在排查过程中发现这些 Tracer 数据会在客户端序列化，放到 header 里。 image

SOFARPC 底层使用的是 SOFABolt ，在 Bolt 中用于传输数据的类是RpcCommand，RpcCommand 类里有一个headerLength字段用来存放 header 的长度。

注意，这个字段是 short 类型的。 image

我们在实践中，发现其中有一些情况上报的数据非常大，就像上面的截图一样，header 数据特别大的情况，short 类型就不够用了，导致变成了负数。

所以，才会在服务端发现Head of request is null or is not map

OrezzerO commented 9 months ago

两个问题:

  1. 业务侧(对于bolt 来说, Tracer 也是业务侧) header 长度太长导致了问题.
  2. bolt 层没有很好处理这个问题, 导致报错不明显.

问题1 需要由业务侧解决 问题2 https://github.com/sofastack/sofa-bolt/pull/281 应该已经修复了,版本是 bolt 1.6.6