search

sofastack / sofa-rpc

SOFARPC is a high-performance, high-extensibility, production-level Java RPC framework.
https://www.sofastack.tech/sofa-rpc/docs/Home
Apache License 2.0
3.81k stars 1.17k forks source link

sofa rpc-com.alipay.sofa.rpc.core.exception.SofaRpcException: com.alipay.remoting.exception.RemotingException #557

Closed mailfly closed 5 years ago

mailfly commented 5 years ago

用sofa rpc来发送大量的日志数据时候，报如下错误，哪里配置不合理么？大部分都是默认的配置，生产的网络有时候有80ms的延时。发送的qps 1K不到是时候经常包netty overflow:,各位有什么建议么？ 为啥这么低的qps就这么多报错，每个请求的数据量在4KB 左右 com.alipay.sofa.rpc.core.exception.SofaRpcException: com.alipay.remoting.exception.RemotingException: Check connection failed for address: Origin url [bolt://10.1.1.1:12200?version=1.0&uniqueId=&timeout=3000&delay=-1&id=rpc-cfg-0&dynamic=true&weight=100&accepts=100000&startTime=1553155493354&appName=log-proxy&serialization=hessian2&pid=1&language=java&rpcVer=50408], Unique key [10.1.1.1:12200]., maybe write overflow! at com.alipay.sofa.rpc.transport.bolt.BoltClientTransport.convertToRpcException(BoltClientTransport.java:373) at com.alipay.sofa.rpc.transport.bolt.BoltClientTransport.oneWaySend(BoltClientTransport.java:302) at com.alipay.sofa.rpc.client.AbstractCluster.doSendMsg(AbstractCluster.java:526) at com.alipay.sofa.rpc.client.AbstractCluster.sendMsg(AbstractCluster.java:485) at com.alipay.sofa.rpc.filter.ConsumerInvoker.invoke(ConsumerInvoker.java:60) at com.alipay.sofa.rpc.filter.RpcReferenceContextFilter.invoke(RpcReferenceContextFilter.java:80) at com.alipay.sofa.rpc.filter.FilterInvoker.invoke(FilterInvoker.java:96) at com.alipay.sofa.rpc.filter.ConsumerExceptionFilter.invoke(ConsumerExceptionFilter.java:37) at com.alipay.sofa.rpc.filter.FilterInvoker.invoke(FilterInvoker.java:96) at com.alipay.sofa.rpc.filter.FilterChain.invoke(FilterChain.java:302) at com.alipay.sofa.rpc.client.AbstractCluster.filterChain(AbstractCluster.java:478) at com.alipay.sofa.rpc.client.FailoverCluster.doInvoke(FailoverCluster.java:66) at com.alipay.sofa.rpc.client.AbstractCluster.invoke(AbstractCluster.java:285) at com.alipay.sofa.rpc.client.ClientProxyInvoker.invoke(ClientProxyInvoker.java:83) at com.xxx.proxy.api.LogServiceApi_proxy_0.send(LogServiceApi_proxy_0.java) at com.xxx.proxyclient.logclient.transport.SofaClientSend.logProxy(SofaClientSend.java:105) at com.xxx.proxyclient.logclient.logback.LogProxySofaAppender.append(LogProxySofaAppender.java:59) at com.xxx.proxyclient.logclient.logback.LogProxySofaAppender.append(LogProxySofaAppender.java:33) at ch.qos.logback.core.UnsynchronizedAppenderBase.doAppend(UnsynchronizedAppenderBase.java:84) at ch.qos.logback.core.spi.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:48) at ch.qos.logback.classic.Logger.appendLoopOnAppenders(Logger.java:270) at ch.qos.logback.classic.Logger.callAppenders(Logger.java:257) at ch.qos.logback.classic.Logger.buildLoggingEventAndAppend(Logger.java:421) at ch.qos.logback.classic.Logger.filterAndLog_0_Or3Plus(Logger.java:383) at ch.qos.logback.classic.Logger.info(Logger.java:579) at com.xxx.xxx.xxx.xxx.api.service.thread.xxxTaskNew.doLastxxx(xxxTaskNew.java:163) at com.xxx.xxx.xxx.xxx.api.service.thread.xxxTaskNew.doFilter(xxxTaskNew.java:419) at com.xxx.xxx.xxx.xxx.api.kafka.oneConsumerMultixxx.ConsumerItemxxx.run(ConsumerItemxxx.java:40) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runxxx(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$xxx.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: com.alipay.remoting.exception.RemotingException: Check connection failed for address: Origin url [bolt://10.1.1.1:12200?version=1.0&uniqueId=&timeout=3000&delay=-1&id=rpc-cfg-0&dynamic=true&weight=100&accepts=100000&startTime=1553155493354&appName=log-proxy&serialization=hessian2&pid=1&language=java&rpcVer=50408], Unique key [10.1.1.1:12200]., maybe write overflow! at com.alipay.remoting.DefaultConnectionManager.check(DefaultConnectionManager.java:407) at com.alipay.remoting.rpc.RpcClientRemoting.oneway(RpcClientRemoting.java:54) at com.alipay.remoting.rpc.RpcClient.oneway(RpcClient.java:231) at com.alipay.sofa.rpc.transport.bolt.BoltClientTransport.doOneWay(BoltClientTransport.java:325) at com.alipay.sofa.rpc.transport.bolt.BoltClientTransport.oneWaySend(BoltClientTransport.java:300)

sofa rpc版本5.4.8

mailfly commented 5 years ago

被这个问题搞了很久，各位有什么好的建议么？thanks

leizhiyuan commented 5 years ago

https://github.com/alipay/sofa-rpc/issues/551 这个issue看一下。可以调整buffer 的检测大小

mailfly commented 5 years ago

@leizhiyuan 谢谢及时的回复，netty高水位低水位配置怎样做个合理的设置呢？ 确认网络应该没有打爆 配置32101024 和64101024，先扩大10倍是否合理？

mailfly commented 5 years ago

*号被自动过滤掉了 :(

mailfly commented 5 years ago

还有问下，高水位跟低水位之间的差值是一般如何设置？ 比如我设置设置为32 1024 10 ，高水位 64 1024 10 ?

leizhiyuan commented 5 years ago

先按照这个排查

解决办法：
建议首先排查网络环境（网卡打满也是一种）
网络环境没问题的情况下，看下是否是流量过大，导致下游处理不过来。，如果是这种，可能意味着，这就是压测的瓶颈了。可以评估下游扩容，性能优化等方式来解决。
如果确实认为64k 也不够用. 可以自行调整

最后一步才是调整。如果调整，可以试一下320k。

mailfly commented 5 years ago

ok ,thanks

khotyn commented 5 years ago

@mailfly 请问你们是在生产环境中使用吗？如果公司允许的话，可以在 https://github.com/alipay/sofastack-doc/issues/42 登记一下信息，方便社区更好地提供服务，😆

mailfly commented 5 years ago

@khotyn 是的，好的，这个如果允许的话，会在上面登记，谢谢~

mailfly commented 5 years ago

经过初步调整扩大4,8,10,16倍后，差不多对于我们来说8倍的参数比较合适点，谢谢了，关闭这个issue了，谢谢支持

WindySummerF commented 2 years ago

您好, 请问一下调节的参数是这几个吗, 我再本地配置了感觉作用不是很明显啊, t的值设置成1和100感觉没什么区别, 再设置方面有什么技巧或者计算方式吗, 非常感谢, @leizhiyuan @mailfly @khotyn int t = 1; RpcConfigs.putValue(RpcOptions.TRANSPORT_BUFFER_SIZE, 1024t); RpcConfigs.putValue(RpcOptions.TRANSPORT_BUFFER_MAX, 10248t); RpcConfigs.putValue(RpcOptions.TRANSPORT_BUFFER_MIN, 10244*t);