search

weibocom / motan

A cross-language remote procedure call(RPC) framework for rapid development of high performance distributed services.
Other
5.9k stars 1.78k forks source link

motan-memoryStatistic这个内存为什么一直涨？ #437

Open 89333367 opened 7 years ago

89333367 commented 7 years ago 
[INFO ][motan-memoryStatistic] 101.53MB of 3616.00 MB (2.8%) used 
[2017-05-17 14:26:25,303][org.apache.zookeeper.ClientCnxn:742]
[DEBUG]Got ping response for sessionid: 0x15be0dc7d2e3289 after 2ms
[2017-05-17 14:26:45,303][org.apache.zookeeper.ClientCnxn:742]
[DEBUG]Got ping response for sessionid: 0x15be0dc7d2e3289 after 2ms
[2017-05-17 14:26:45,483][serviceStatsLog:95]
[INFO ][motan-totalAccessStatistic] app: motan module: motan total_count: 0 slow_count: 0 biz_excp: 0 other_excp: 0 avg_time: 0.00ms biz_time: 0.00ms avg_tps: 0
[2017-05-17 14:26:45,483][serviceStatsLog:99]
[INFO ][motan-memoryStatistic] 101.53MB of 3616.00 MB (2.8%) used 
[2017-05-17 14:27:05,304][org.apache.zookeeper.ClientCnxn:742]
[DEBUG]Got ping response for sessionid: 0x15be0dc7d2e3289 after 2ms
[2017-05-17 14:27:15,483][serviceStatsLog:95]
[INFO ][motan-totalAccessStatistic] app: motan module: motan total_count: 0 slow_count: 0 biz_excp: 0 other_excp: 0 avg_time: 0.00ms biz_time: 0.00ms avg_tps: 0
[2017-05-17 14:27:15,483][serviceStatsLog:99]
[INFO ][motan-memoryStatistic] 101.53MB of 3616.00 MB (2.8%) used

到99了就挂了

rayzhang0603 commented 7 years ago

memoryStatistic只是统计了jvm内存使用情况，具体见StatsUtil.logMemoryStatistic() 内存一直增加应该是其他原因导致,建议dump堆分析一下

xiaoxiaoyumi commented 6 years ago

我在用的时候，也是内存一直在涨，占用很快，本地项目启动的时候，也是一样的。

rayzhang0603 commented 6 years ago

@xiaoxiaoyumi 内存持续增长可能跟业务使用姿势或者业务逻辑有关，可以通过jmap分析一下堆内存对象占用情况。

89333367 commented 6 years ago

这个我一直都没有找到原因，然后我就切换到了dubbo上面。内存就没有增长过。代码逻辑是一样的。

xiaoxiaoyumi commented 6 years ago

我这个项目太简单了，目前应用没有什么占用的，dump文件下载了没有任何问题，目前还是持续在增长

xiaoxiaoyumi commented 6 years ago

目前我用motan 只要是为了提供给php应用，使用yar协议这个，所以如果找不到原因解决这个问题，只能换成dubbo了，今天研究一下源码吧

rayzhang0603 commented 6 years ago

@xiaoxiaoyumi 可以确认一下dump文件中实际占用内存有没有变大？如果变大是哪类对象增加了？如果堆内内存没有变大，是实际进程占用的内存变大了（堆外内存）？

xiaoxiaoyumi commented 6 years ago

没有任何问题，采用的饿druid 连接池的dump，况且就三个支付通道，都是提供rest接口，目前没有上线，本地项目启动，增长速度很快 [motan-memoryStatistic] 257.38MB of 1806.50 MB (14.2%) used
2018-07-17 13:50:25,677 INFO [pool-5-thread-1] serviceStatsLog - [motan-statisticCallback] identity: motan://172.16.100.43:8001/java_config//1.0/service connectionCount: 0 taskCount: 0 queueCount: 0 maxThreadCount: 800 maxTaskCount: 800 2018-07-17 13:50:55,676 INFO [pool-5-thread-1] serviceStatsLog - [motan-totalAccessStatistic] app: motan module: motan total_count: 0 slow_count: 0 biz_excp: 0 other_excp: 0 avg_time: 0.00ms biz_time: 0.00ms avg_tps: 0 2018-07-17 13:50:55,676 INFO [pool-5-thread-1] serviceStatsLog - [motan-memoryStatistic] 269.48MB of 1806.50 MB (14.9%) used
2018-07-17 13:50:55,677 INFO [pool-5-thread-1] serviceStatsLog - [motan-statisticCallback] identity: motan://172.16.100.43:8001/

xiaoxiaoyumi commented 6 years ago

总觉得，你们应该有一个参数，但是看了wiki上你们开源的参数配置，基本上除了一个accessLog，没有其他的，这就尴尬了~

rayzhang0603 commented 6 years ago

@xiaoxiaoyumi motan的这个统计只是获取了jvm的堆内存占用情况，并不是指motan所占用的内存，打印或者不打印日志都不会对导致内存问题。

可以用jmap查看一下堆内具体是哪种对象在持续增加，占用内存。

xiaoxiaoyumi commented 6 years ago

ok

xiaoxiaoyumi commented 6 years ago

RefererInvocationHandler invoke Error: uri=motan://10.31.24.33:0/cn.yp.service.SupportRuleService requestId=1609557160433287171 interface=cn.yp.service.SupportRuleService method=getRuleResult(cn.yp.entity.IntoParamDO) com.weibo.api.motan.exception.MotanServiceException: error_message: NettyChannel send request to server Error: url=motan://10.31.24.33:8001/cn.yp.service.SupportRuleService local=/10.31.24.33:39662 requestId=1609557160433287171 interface=cn.yp.service.SupportRuleService method=getRuleResult(cn.yp.entity.IntoParamDO), status: 503, error_code: 10001,r=null at com.weibo.api.motan.transport.netty.NettyChannel.request(NettyChannel.java:96) at com.weibo.api.motan.transport.netty.NettyClient.request(NettyClient.java:163) at com.weibo.api.motan.transport.netty.NettyClient.request(NettyClient.java:110) at com.weibo.api.motan.protocol.rpc.DefaultRpcReferer.doCall(DefaultRpcReferer.java:52) at com.weibo.api.motan.rpc.AbstractReferer.call(AbstractReferer.java:64) at com.weibo.api.motan.cluster.ha.FailoverHaStrategy.call(FailoverHaStrategy.java:72) at com.weibo.api.motan.cluster.support.ClusterSpi.call(ClusterSpi.java:73) at com.weibo.api.motan.proxy.AbstractRefererHandler.invokeRequest(AbstractRefererHandler.java:76) at com.weibo.api.motan.proxy.RefererInvocationHandler.invoke(RefererInvocationHandler.java:72) at com.sun.proxy.$Proxy121.getRuleResult(Unknown Source) at cn.yp.into.resource.rules.impl.RuleParamterServiceImpl.ruleParamterVerify(RuleParamterServiceImpl.java:50) at cn.yp.into.resource.impl.UserOcrVerifyServiceImpl.getOcrMessage(UserOcrVerifyServiceImpl.java:104) at cn.yp.into.api.web.authentication.UserOcrVerifyApiController.verifyOcrMessage(UserOcrVerifyApiController.java:102) at sun.reflect.GeneratedMethodAccessor266.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205) at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133) at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97) at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827) at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738) at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85) at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967) at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901) at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970) at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:872) at javax.servlet.http.HttpServlet.service(HttpServlet.java:661) at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846) at javax.servlet.http.HttpServlet.service(HttpServlet.java:742) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at com.alibaba.druid.support.http.WebStatFilter.doFilter(WebStatFilter.java:123) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:197) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.springframework.boot.web.filter.ApplicationContextHeaderFilter.doFilterInternal(ApplicationContextHeaderFilter.java:55) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.springframework.boot.actuate.trace.WebRequestTraceFilter.doFilterInternal(WebRequestTraceFilter.java:110) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.springframework.web.filter.HttpPutFormContentFilter.doFilterInternal(HttpPutFormContentFilter.java:108) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:81) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:197) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.springframework.boot.actuate.autoconfigure.MetricsFilter.doFilterInternal(MetricsFilter.java:106) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.springframework.boot.web.support.ErrorPageFilter.doFilter(ErrorPageFilter.java:115) at org.springframework.boot.web.support.ErrorPageFilter.access$000(ErrorPageFilter.java:59) at org.springframework.boot.web.support.ErrorPageFilter$1.doFilterInternal(ErrorPageFilter.java:90) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) at org.springframework.boot.web.support.ErrorPageFilter.doFilter(ErrorPageFilter.java:108) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:198) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:478) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:80) at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:650) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342) at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:799) at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66) at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:868) at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1457) at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.lang.Thread.run(Thread.java:748) Caused by: java.nio.channels.ClosedChannelException: null at org.jboss.netty.channel.socket.nio.AbstractNioWorker.cleanUpWriteBuffer(AbstractNioWorker.java:434) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromUserCode(AbstractNioWorker.java:129) at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:84) at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:779) at org.jboss.netty.channel.Channels.write(Channels.java:725) at org.jboss.netty.handler.codec.oneone.OneToOneEncoder.doEncode(OneToOneEncoder.java:71) at org.jboss.netty.handler.codec.oneone.OneToOneEncoder.handleDownstream(OneToOneEncoder.java:59) at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:591) at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:784) at org.jboss.netty.channel.SimpleChannelHandler.writeRequested(SimpleChannelHandler.java:292) at org.jboss.netty.channel.SimpleChannelHandler.handleDownstream(SimpleChannelHandler.java:254) at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:591) at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:582) at org.jboss.netty.channel.Channels.write(Channels.java:704) at org.jboss.netty.channel.Channels.write(Channels.java:671) at org.jboss.netty.channel.AbstractChannel.write(AbstractChannel.java:248) at com.weibo.api.motan.transport.netty.NettyChannel.request(NettyChannel.java:65) ... 92 common frames omitted

测试环境全是这个问题，到目前为止，没有找到解决方案~

changyu.qi@hotmail.com

发件人： Raymailto:notifications@github.com 发送时间： 2018-07-17 14:06 收件人： weibocom/motanmailto:motan@noreply.github.com 抄送： xiaoxiaoyumimailto:changyu.qi@hotmail.com; Mentionmailto:mention@noreply.github.com 主题： Re: [weibocom/motan] motan-memoryStatistic这个内存为什么一直涨？ (#437)

@xiaoxiaoyumihttps://github.com/xiaoxiaoyumi motan的这个统计只是获取了jvm的堆内存占用情况，并不是指motan所占用的内存，打印或者不打印日志都不会对导致内存问题。

可以用jmap查看一下堆内具体是哪种对象在持续增加，占用内存。

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/weibocom/motan/issues/437#issuecomment-405469958, or mute the threadhttps://github.com/notifications/unsubscribe-auth/Affh2uKmQHPKxzvwnzIqIPbWg0tgbf3gks5uHX7ogaJpZM4NdbSg.

rayzhang0603 commented 6 years ago

Caused by: java.nio.channels.ClosedChannelException 这个请求没有发送出去，链接已经关闭了，可以看看这个异常之前是否有其他类型的异常，例如大量超时之类的异常

qichangyu commented 6 years ago

解决了，上周五，就是因为开启心跳问题，在spring 加载完成以后，再开启就可以了~另外当时是直连的问题，后来心跳调整完成以后，zk上面也注册成功了，这边也就解决了，很感谢~