search

apache / mina-sshd

Apache MINA sshd is a comprehensive Java library for client- and server-side SSH.
https://mina.apache.org/sshd-project/
Apache License 2.0
912 stars 361 forks source link

The remote port forwarding on the server cannot be released, causing a large number of CLOSE_WAIT states. #632

Open wumengchao opened 1 week ago

wumengchao commented 1 week ago

Version

2.14.0

Bug description

Discovered an issue that is difficult to reproduce and troubleshoot. The phenomenon is as follows: After creating port forwarding on the client through SSH -R, the client's network disconnects for uncertain reasons. However, the server's port cannot be released, and when testing requests to the server's port, a large number of CLOSE_WAITs appear.

Actual behavior

After creating port forwarding on the client through SSH -R, the client's network disconnects for uncertain reasons. However, the server's port cannot be released, and when testing requests to the server's port, a large number of CLOSE_WAITs appear.

Expected behavior

Hope to be provided with troubleshooting methods, and ideally a quick resolution.

Relevant log output

No response

Other information

No response

wumengchao commented 6 days ago

WARN o.apache.sshd.netty.NettyIoSession - channelActive(session=NettyIoSession[local=/127.0.0.1:8443, remote=/127.0.0.1:5023]): could not create SSH session (java.lang.IllegalStateException); closing java.lang.IllegalStateException: Channel id=82 not registered because session is being closed: TcpipClientChannel[id=82, recipient=-1]-ServerSessionImpl[deploy@/127.0.0.1:3471] at org.apache.sshd.common.channel.AbstractChannel.handleChannelRegistrationResult(AbstractChannel.java:471) at org.apache.sshd.common.session.helpers.AbstractConnectionService.registerChannel(AbstractConnectionService.java:424) at org.apache.sshd.common.forward.DefaultForwarder$StaticIoHandler.sessionCreated(DefaultForwarder.java:1024) at org.apache.sshd.netty.NettyIoSession.channelActive(NettyIoSession.java:254) at org.apache.sshd.netty.NettyIoSession$Adapter.channelActive(NettyIoSession.java:356) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:260) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:238) at io.netty.channel.AbstractChannelHandlerContext.fireChannelActive(AbstractChannelHandlerContext.java:231) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelActive(DefaultChannelPipeline.java:1395) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:258) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:238) at io.netty.channel.DefaultChannelPipeline.fireChannelActive(DefaultChannelPipeline.java:894) at io.netty.channel.AbstractChannel$AbstractUnsafe.register0(AbstractChannel.java:521) at io.netty.channel.AbstractChannel$AbstractUnsafe.access$200(AbstractChannel.java:428) at io.netty.channel.AbstractChannel$AbstractUnsafe$1.run(AbstractChannel.java:485) at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173) at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166) at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:469) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:569) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:994) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Thread.java:1570)

wumengchao commented 6 days ago

Port 8443 process has ended, but the port forwarded by mina-sshd is still occupied and cannot be released.

tomaswolf commented 6 days ago

Can you draw me a picture similar to the ones in https://github.com/apache/mina-sshd/blob/master/docs/technical/tcpip-forwarding.md ? Which server corresponds to "whatever", which port is "A" and "B", and what is your exact ssh command?

The exception message says the session was being closed. So why is it being closed? There must be some earlier problem. Can you run that server with debug logging?

Finally, I see the server is using the Netty transport back-end. Can you try running it with the NIO2 back-end instead? Does the problem also exist with NIO2?