search

sofastack / sofa-bolt

SOFABolt is a lightweight, easy to use and high performance remoting framework based on Netty.
https://www.sofastack.tech/projects/sofa-bolt/
Apache License 2.0
2.44k stars 860 forks source link

运行一段时间就会报ClosedChannelException错误 #266

Closed willardwang closed 2 years ago

willardwang commented 2 years ago

Describe the bug A clear and concise description of what the bug is. 大概运行一两周就会报错： 2021-12-06 11:09:09--Rpc-netty-server-worker-1-thread-12--Rpc response send failed! id=2305369. The address is 10.10.10.24:39600 java.nio.channels.ClosedChannelException: null at io.netty.channel.AbstractChannel$AbstractUnsafe.write(...)(Unknown Source) ~[netty-all-4.1.32.Final.jar:4.1.32.Final] 奇怪的是没有详细堆栈

Expected behavior

Actual behavior

Steps to reproduce

Minimal yet complete reproducer code (or GitHub URL to code)

Environment

willardwang commented 2 years ago

麻烦大神们帮忙看看 </抱拳>

willardwang commented 2 years ago

日志 bolt-error.log

seeflood commented 2 years ago

感谢反馈 ! @dbl-x @ xmtsui 帮忙看下哈

willardwang commented 2 years ago

生产环境问题，拜托了🙏 @seeflood @dbl-x @xmtsui

willardwang commented 2 years ago

好像是因为client端确实关闭了连接，client端检测到server端反复加入再丢失

seeflood commented 2 years ago

相关技术栈是啥？（比如用的是SOFA-RPC、SOFA-Boot还是别的，RPC和Boot版本是啥） 可以多提供些信息，我明天线下找各位owner请教下~

willardwang commented 2 years ago

没使用SOFA-RPC、SOFA-Boot等sofa相关技术栈，sofa相关的只有一个sofa-common-tools-1.0.12.jar，netty是netty-all-4.1.32.Final.jar

dbl-x commented 2 years ago

2021-12-06 11:09:09--Rpc-netty-server-worker-1-thread-11--Rpc response send failed! id=2202224. The address is 10.10.10.23:51294 java.nio.channels.ClosedChannelException: null

看了日志就这这一行，只是说给响应的时候10.10.10.23:51294这个连接已经中断了，你可以看看当时10.10.10.23机器上的日志，看看是否连接中断了。另外可以看看Bolt的connection-event日志，看看客户端和服务端的建连日志。

willardwang commented 2 years ago

已经在钉钉上联系开发者了，超赞👍 开启了bolt日志，再出问题的时候提供具体日志。

willardwang commented 2 years ago

已经解决了，是我们系统bug导致的gc停顿，现象是节点丢失，感谢开发者团队的支持 谢谢啦