search

sohutv / mqcloud

RocketMQ企业级一站式服务平台
Apache License 2.0
1.91k stars 273 forks source link

客户端在生产消息时偶尔报发送失败报错，报错信息见正文 #51

Open demonhuahua opened 1 year ago

demonhuahua commented 1 year ago

2023-11-21 07:56:56.709 ERROR [DubboServerHandler-172.20.3.143:58439-thread-199] [TID:1b4771c8afc94ee7b46c5a9b28679de4.620.17005246136686327] c.sohu.tv.mq.rocketmq.RocketMQProducer Line:284 - send error 在生产消息时，偶尔报如上错误，在故障点，排查rocketmq broker均无资源凸峰，但是有如下error。

2023-11-21 07:57:08 ERROR NettyServerNIOSelector_3_2 - processRequestWrapper response to /172.16.14.111:24829 failed io.netty.channel.StacklessClosedChannelException: null ----->这个error，我估计是不是客户端发送失败关闭了连接，因此broker返回数据失败。

烦请给个send error报错的解决思路。

gaoyf commented 1 year ago

可以贴出报错的异常堆栈，还有mqcloud及broker版本发一下

demonhuahua commented 1 year ago

broker版本：V4_9_1 mqcloud版本：mq-cloud-4.9.3.1

2023-11-21 07:57:00.168 ERROR [DubboServerHandler-172.20.30.111:17690-thread-193] [TID:e66dcae9549c46b687e51d0ae3f7695a.570.17005246170697275] c.sohu.tv.mq.rocketmq.RocketMQProducer Line:284 - send error org.apache.rocketmq.remoting.exception.RemotingTooMuchRequestException: sendDefaultImpl call timeout at org.apache.rocketmq.client.impl.producer.DefaultMQProducerImpl.sendDefaultImpl(DefaultMQProducerImpl.java:690) at org.apache.rocketmq.client.impl.producer.DefaultMQProducerImpl.send(DefaultMQProducerImpl.java:1398) at org.apache.rocketmq.client.impl.producer.DefaultMQProducerImpl.send(DefaultMQProducerImpl.java:1342) at org.apache.rocketmq.client.producer.DefaultMQProducer.send(DefaultMQProducer.java:330) at com.sohu.tv.mq.rocketmq.RocketMQProducer.publish(RocketMQProducer.java:270) at com.sohu.tv.mq.rocketmq.RocketMQProducer.publish(RocketMQProducer.java:259) at com.sohu.tv.mq.rocketmq.RocketMQProducer.publish(RocketMQProducer.java:219) at com.sohu.tv.mq.rocketmq.RocketMQProducer.publish(RocketMQProducer.java:207) at com.sohu.tv.mq.rocketmq.RocketMQProducer.publish(RocketMQProducer.java:181) at com.sohu.tv.mq.rocketmq.RocketMQProducer$$FastClassBySpringCGLIB$$8e6b35ea.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:687) at com.sohu.tv.mq.rocketmq.RocketMQProducer$$EnhancerBySpringCGLIB$$aa0559af.publish() at net.aukid.message.service.PushService.sendMessagePushMq(PushService.java:78) at net.aukid.message.service.PushService$$FastClassBySpringCGLIB$$256db20e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:771) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749) at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:100) at net.aukid.message.aop.DubboExceptionAspect.doAround(DubboExceptionAspect.java:52) at sun.reflect.GeneratedMethodAccessor388.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:644) at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:633) at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:70) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:95) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:691) at net.aukid.message.service.PushService$$EnhancerBySpringCGLIB$$975926da.sendMessagePushMq() at org.apache.dubbo.common.bytecode.Wrapper15.invokeMethod(Wrapper15.java) at org.apache.dubbo.rpc.proxy.javassist.JavassistProxyFactory$1.doInvoke(JavassistProxyFactory.java:47) at org.apache.dubbo.rpc.proxy.AbstractProxyInvoker.invoke(AbstractProxyInvoker.java:84) at org.apache.dubbo.config.invoker.DelegateProviderMetaDataInvoker.invoke(DelegateProviderMetaDataInvoker.java:56) at org.apache.dubbo.rpc.protocol.InvokerWrapper.invoke(InvokerWrapper.java:56) at org.apache.dubbo.monitor.support.MonitorFilter.invoke$original$XpHr6BjT(MonitorFilter.java:89) at org.apache.dubbo.monitor.support.MonitorFilter.invoke$original$XpHr6BjT$accessor$oYaHBMqy(MonitorFilter.java) at org.apache.dubbo.monitor.support.MonitorFilter$auxiliary$BwC4Vmnn.call(Unknown Source) at org.apache.skywalking.apm.agent.core.plugin.interceptor.enhance.InstMethodsInter.intercept(InstMethodsInter.java:86) at org.apache.dubbo.monitor.support.MonitorFilter.invoke(MonitorFilter.java) at org.apache.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:81) at org.apache.dubbo.rpc.filter.TimeoutFilter.invoke(TimeoutFilter.java:44) at org.apache.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:81) at org.apache.dubbo.rpc.protocol.dubbo.filter.TraceFilter.invoke(TraceFilter.java:81) at org.apache.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:81) at org.apache.dubbo.rpc.filter.ContextFilter.invoke(ContextFilter.java:102) at org.apache.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:81) at org.apache.dubbo.rpc.filter.GenericFilter.invoke(GenericFilter.java:149) at org.apache.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:81) at org.apache.dubbo.rpc.filter.ClassLoaderFilter.invoke(ClassLoaderFilter.java:38) at org.apache.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:81) at org.apache.dubbo.rpc.filter.EchoFilter.invoke(EchoFilter.java:41) at org.apache.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:81) at org.apache.dubbo.rpc.protocol.dubbo.DubboProtocol$1.reply(DubboProtocol.java:150) at org.apache.dubbo.remoting.exchange.support.header.HeaderExchangeHandler.handleRequest(HeaderExchangeHandler.java:100) at org.apache.dubbo.remoting.exchange.support.header.HeaderExchangeHandler.received(HeaderExchangeHandler.java:175) at org.apache.dubbo.remoting.transport.DecodeHandler.received(DecodeHandler.java:51) at org.apache.dubbo.remoting.transport.dispatcher.ChannelEventRunnable.run(ChannelEventRunnable.java:57) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)

gaoyf commented 1 year ago
  1. 你分析的broker响应失败应该是对的。
  2. 应该是客户端应用的问题。
  3. 是否还有其他异常？如果方便的话可以上传rocketmq日志文件（目前日志太少，看不出端倪）。 具体分析如下，就目前的异常栈来看，对应rocketmq4.9.1版本代码如下： image 如果该异常前后日志中无其他异常日志，根据上面的代码，异常是由于直接callTimeout跳出，而没有远程请求耗时操作，那么建议你分析下应用的gc情况和进入安全点的耗时。 可以添加如下参数开启相关记录： gc参数：-Xloggc:/应用目录/logs/gc.log -XX:+PrintGCDateStamps 安全点参数：-XX:+PrintSafepointStatistics -XX:PrintSafepointStatisticsCount=1 -XX:+SafepointTimeout -XX:SafepointTimeoutDelay=500 具体结果含义请自行搜索。