search

alibaba / arthas

Alibaba Java Diagnostic Tool Arthas/Alibaba Java诊断利器Arthas
https://arthas.aliyun.com/
Apache License 2.0
35.55k stars 7.47k forks source link

用tunnel-server连接springboot的arthas一段时间后报Can not find arthas agent by id #1573

Closed Ronaltn closed 3 years ago

Ronaltn commented 3 years ago

环境信息

重现问题的步骤

  1. springboot应用中在pom.xml里添加arthas-spring-boot-starter依赖，在application.yaml里添加配置arthas.agent-id和arthas.tunnel-server；
  2. 部署tunnel-server，刚启动是可以连的，但过一段时间再用tunnel-server的webconsole就会报：Can not find arthas agent by id:xxx。

实际运行的结果

springboot应用中没有日志输出，tunnel-server有日志。

2020-11-09 09:06:28.171  INFO 1 --- [Server-boss-1-1] io.netty.handler.logging.LoggingHandler  : [id: 0x4472dca3, L:/0.0.0.0:7777] READ: [id: 0xc628850b, L:/172.20.0.241:7777 - R:/172.20.0.1:2200]
2020-11-09 09:06:28.171  INFO 1 --- [Server-boss-1-1] io.netty.handler.logging.LoggingHandler  : [id: 0x4472dca3, L:/0.0.0.0:7777] READ COMPLETE
2020-11-09 09:06:28.174  INFO 1 --- [rver-worker-3-2] c.a.a.t.server.TunnelSocketFrameHandler  : websocket handshake complete, uri: /ws?method=connectArthas&id=172.20.0.7
2020-11-09 09:06:28.175  INFO 1 --- [rver-worker-3-2] c.a.a.t.server.TunnelSocketFrameHandler  : try to connect to arthas agent, id: 172.20.0.7
2020-11-09 09:06:28.175  INFO 1 --- [rver-worker-3-2] c.a.a.t.server.TunnelSocketFrameHandler  : random clientConnectionId: 7NJ74AXM8FRBLDNSSHRG
2020-11-09 09:06:28.175  INFO 1 --- [rver-worker-3-2] c.a.a.t.server.TunnelSocketFrameHandler  : startTunnel response: response:/?method=startTunnel&id=172.20.0.7&clientConnectionId=7NJ74AXM8FRBLDNSSHRG
2020-11-09 09:06:28.175  INFO 1 --- [rver-worker-3-2] c.a.a.t.server.TunnelSocketFrameHandler  : browser connect waitting for arthas agent open tunnel
2020-11-09 09:06:28.177  WARN 1 --- [rver-worker-3-1] io.netty.channel.DefaultChannelPipeline  : An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.

java.io.IOException: Connection reset by peer
    at sun.nio.ch.FileDispatcherImpl.read0(Native Method) ~[na:1.8.0_265]
    at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) ~[na:1.8.0_265]
    at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) ~[na:1.8.0_265]
    at sun.nio.ch.IOUtil.read(IOUtil.java:192) ~[na:1.8.0_265]
    at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:377) ~[na:1.8.0_265]
    at io.netty.buffer.PooledByteBuf.setBytes(PooledByteBuf.java:253) ~[netty-buffer-4.1.46.Final.jar!/:4.1.46.Final]
    at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1133) ~[netty-buffer-4.1.46.Final.jar!/:4.1.46.Final]
    at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:350) ~[netty-transport-4.1.46.Final.jar!/:4.1.46.Final]
    at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:148) ~[netty-transport-4.1.46.Final.jar!/:4.1.46.Final]
    at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) [netty-transport-4.1.46.Final.jar!/:4.1.46.Final]
    at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650) [netty-transport-4.1.46.Final.jar!/:4.1.46.Final]
    at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576) [netty-transport-4.1.46.Final.jar!/:4.1.46.Final]
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [netty-transport-4.1.46.Final.jar!/:4.1.46.Final]
    at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-common-4.1.46.Final.jar!/:4.1.46.Final]
    at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.46.Final.jar!/:4.1.46.Final]
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-common-4.1.46.Final.jar!/:4.1.46.Final]
    at java.lang.Thread.run(Thread.java:748) [na:1.8.0_265]

2020-11-09 09:06:48.176 ERROR 1 --- [rver-worker-3-2] c.a.a.t.server.TunnelSocketFrameHandler  : browser connect wait for arthas agent open tunnel timeout, agentId: [172.20.0.7], clientConnectionId: 7NJ74AXM8FRBLDNSSHRG
2020-11-09 09:20:35.995  INFO 1 --- [Server-boss-1-1] io.netty.handler.logging.LoggingHandler  : [id: 0x4472dca3, L:/0.0.0.0:7777] READ: [id: 0xadb361e3, L:/172.20.0.241:7777 - R:/172.20.0.216:51301]
2020-11-09 09:20:35.995  INFO 1 --- [Server-boss-1-1] io.netty.handler.logging.LoggingHandler  : [id: 0x4472dca3, L:/0.0.0.0:7777] READ COMPLETE
2020-11-09 09:20:36.037  INFO 1 --- [rver-worker-3-1] c.a.a.t.server.TunnelSocketFrameHandler  : websocket handshake complete, uri: /ws?method=connectArthas&id=172.20.0.7
2020-11-09 09:20:36.037  INFO 1 --- [rver-worker-3-1] c.a.a.t.server.TunnelSocketFrameHandler  : try to connect to arthas agent, id: 172.20.0.7
2020-11-09 09:20:36.038 ERROR 1 --- [rver-worker-3-1] c.a.a.t.server.TunnelSocketFrameHandler  : Can not find arthas agent by id: [172.20.0.7]
2020-11-09 09:20:36.039  WARN 1 --- [rver-worker-3-1] io.netty.channel.DefaultChannelPipeline  : An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.

java.lang.IllegalArgumentException: Can not find arthas agent by id: [172.20.0.7]
    at com.alibaba.arthas.tunnel.server.TunnelSocketFrameHandler.connectArthas(TunnelSocketFrameHandler.java:205) ~[classes!/:3.4.4]
    at com.alibaba.arthas.tunnel.server.TunnelSocketFrameHandler.userEventTriggered(TunnelSocketFrameHandler.java:68) ~[classes!/:3.4.4]
    at io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:344) [netty-transport-4.1.46.Final.jar!/:4.1.46.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:330) [netty-transport-4.1.46.Final.jar!/:4.1.46.Final]
    at io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:322) [netty-transport-4.1.46.Final.jar!/:4.1.46.Final]
    at io.netty.handler.codec.http.websocketx.WebSocketServerProtocolHandshakeHandler$1.operationComplete(WebSocketServerProtocolHandshakeHandler.java:103) [netty-codec-http-4.1.46.Final.jar!/:4.1.46.Final]
    at io.netty.handler.codec.http.websocketx.WebSocketServerProtocolHandshakeHandler$1.operationComplete(WebSocketServerProtocolHandshakeHandler.java:92) [netty-codec-http-4.1.46.Final.jar!/:4.1.46.Final]
    at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:577) [netty-common-4.1.46.Final.jar!/:4.1.46.Final]
    at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:551) [netty-common-4.1.46.Final.jar!/:4.1.46.Final]
    at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:490) [netty-common-4.1.46.Final.jar!/:4.1.46.Final]
    at io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:183) [netty-common-4.1.46.Final.jar!/:4.1.46.Final]
    at io.netty.channel.DefaultChannelPromise.addListener(DefaultChannelPromise.java:95) [netty-transport-4.1.46.Final.jar!/:4.1.46.Final]
    at io.netty.channel.DefaultChannelPromise.addListener(DefaultChannelPromise.java:30) [netty-transport-4.1.46.Final.jar!/:4.1.46.Final]
    at io.netty.handler.codec.http.websocketx.WebSocketServerProtocolHandshakeHandler.channelRead(WebSocketServerProtocolHandshakeHandler.java:92) [netty-codec-http-4.1.46.Final.jar!/:4.1.46.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377) [netty-transport-4.1.46.Final.jar!/:4.1.46.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) [netty-transport-4.1.46.Final.jar!/:4.1.46.Final]
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355) [netty-transport-4.1.46.Final.jar!/:4.1.46.Final]
    at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) [netty-transport-4.1.46.Final.jar!/:4.1.46.Final]
    at io.netty.handler.codec.http.websocketx.extensions.WebSocketServerExtensionHandler.channelRead(WebSocketServerExtensionHandler.java:101) [netty-codec-http-4.1.46.Final.jar!/:4.1.46.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377) [netty-transport-4.1.46.Final.jar!/:4.1.46.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) [netty-transport-4.1.46.Final.jar!/:4.1.46.Final]
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355) [netty-transport-4.1.46.Final.jar!/:4.1.46.Final]
    at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.50.Final.jar!/:4.1.50.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377) [netty-transport-4.1.46.Final.jar!/:4.1.46.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) [netty-transport-4.1.46.Final.jar!/:4.1.46.Final]
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355) [netty-transport-4.1.46.Final.jar!/:4.1.46.Final]
    at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436) [netty-transport-4.1.46.Final.jar!/:4.1.46.Final]
    at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324) [netty-codec-4.1.50.Final.jar!/:4.1.50.Final]
    at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:296) [netty-codec-4.1.50.Final.jar!/:4.1.50.Final]
    at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251) [netty-transport-4.1.46.Final.jar!/:4.1.46.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377) [netty-transport-4.1.46.Final.jar!/:4.1.46.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) [netty-transport-4.1.46.Final.jar!/:4.1.46.Final]
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355) [netty-transport-4.1.46.Final.jar!/:4.1.46.Final]
    at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.46.Final.jar!/:4.1.46.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377) [netty-transport-4.1.46.Final.jar!/:4.1.46.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) [netty-transport-4.1.46.Final.jar!/:4.1.46.Final]
    at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.46.Final.jar!/:4.1.46.Final]
    at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.46.Final.jar!/:4.1.46.Final]
    at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) [netty-transport-4.1.46.Final.jar!/:4.1.46.Final]
    at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650) [netty-transport-4.1.46.Final.jar!/:4.1.46.Final]
    at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576) [netty-transport-4.1.46.Final.jar!/:4.1.46.Final]
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [netty-transport-4.1.46.Final.jar!/:4.1.46.Final]
    at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-common-4.1.46.Final.jar!/:4.1.46.Final]
    at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.46.Final.jar!/:4.1.46.Final]
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-common-4.1.46.Final.jar!/:4.1.46.Final]
    at java.lang.Thread.run(Thread.java:748) [na:1.8.0_265]

2020-11-09 09:20:36.131  WARN 1 --- [rver-worker-3-1] io.netty.channel.DefaultChannelPipeline  : An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.

io.netty.handler.codec.http.websocketx.CorruptedWebSocketFrameException: Invalid close frame getStatus code: 2000
    at io.netty.handler.codec.http.websocketx.WebSocket08FrameDecoder.protocolViolation(WebSocket08FrameDecoder.java:426) ~[netty-codec-http-4.1.46.Final.jar!/:4.1.46.Final]
    at io.netty.handler.codec.http.websocketx.WebSocket08FrameDecoder.protocolViolation(WebSocket08FrameDecoder.java:422) ~[netty-codec-http-4.1.46.Final.jar!/:4.1.46.Final]
    at io.netty.handler.codec.http.websocketx.WebSocket08FrameDecoder.checkCloseFrameBody(WebSocket08FrameDecoder.java:479) ~[netty-codec-http-4.1.46.Final.jar!/:4.1.46.Final]
    at io.netty.handler.codec.http.websocketx.WebSocket08FrameDecoder.decode(WebSocket08FrameDecoder.java:339) ~[netty-codec-http-4.1.46.Final.jar!/:4.1.46.Final]
    at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:501) ~[netty-codec-4.1.50.Final.jar!/:4.1.50.Final]
    at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:440) ~[netty-codec-4.1.50.Final.jar!/:4.1.50.Final]
    at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) ~[netty-codec-4.1.50.Final.jar!/:4.1.50.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377) [netty-transport-4.1.46.Final.jar!/:4.1.46.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) [netty-transport-4.1.46.Final.jar!/:4.1.46.Final]
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355) [netty-transport-4.1.46.Final.jar!/:4.1.46.Final]
    at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.46.Final.jar!/:4.1.46.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377) [netty-transport-4.1.46.Final.jar!/:4.1.46.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) [netty-transport-4.1.46.Final.jar!/:4.1.46.Final]
    at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.46.Final.jar!/:4.1.46.Final]
    at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.46.Final.jar!/:4.1.46.Final]
    at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) [netty-transport-4.1.46.Final.jar!/:4.1.46.Final]
    at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650) [netty-transport-4.1.46.Final.jar!/:4.1.46.Final]
    at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576) [netty-transport-4.1.46.Final.jar!/:4.1.46.Final]
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [netty-transport-4.1.46.Final.jar!/:4.1.46.Final]
    at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-common-4.1.46.Final.jar!/:4.1.46.Final]
    at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.46.Final.jar!/:4.1.46.Final]
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-common-4.1.46.Final.jar!/:4.1.46.Final]
    at java.lang.Thread.run(Thread.java:748) [na:1.8.0_265]
hengyunabc commented 3 years ago

检查client端的日志，看有没有异常。 ~/logs/arthas/arthas.log 。另外，在client机器上，检查进程是否还活着，检查网络连接是否有连到server的。 client默认是会自动重连的。

Ronaltn commented 3 years ago

检查client端的日志，看有没有异常。 ~/logs/arthas/arthas.log 。另外，在client机器上，检查进程是否还活着，检查网络连接是否有连到server的。 client默认是会自动重连的。

不使用tunnel-server，直接用浏览器打开连到springboot应用的3658端口，是可以正常使用的。 以下是~/logs/arthas/arthas.log的内容：

2020-11-09 09:09:56 [arthas-NettyHttpTelnetBootstrap-4-1] INFO  c.a.a.d.i.n.h.logging.LoggingHandler -[id: 0xe59f20c5, L:/172.20.0.7:3658] READ: [id: 0xb1863c8e, L:/172.20.0.7:3658 - R:/172.20.0.32:55426]
2020-11-09 09:09:56 [arthas-NettyHttpTelnetBootstrap-4-1] INFO  c.a.a.d.i.n.h.logging.LoggingHandler -[id: 0xe59f20c5, L:/172.20.0.7:3658] READ COMPLETE
2020-11-09 09:10:26 [arthas-NettyWebsocketTtyBootstrap-5-1] INFO  c.a.a.d.i.n.h.logging.LoggingHandler -[id: 0xa953fdd9, L:/172.20.0.7:8563] READ: [id: 0xaf8733e8, L:/172.20.0.7:8563 - R:/172.20.0.32:59218]
2020-11-09 09:10:26 [arthas-NettyWebsocketTtyBootstrap-5-1] INFO  c.a.a.d.i.n.h.logging.LoggingHandler -[id: 0xa953fdd9, L:/172.20.0.7:8563] READ COMPLETE
2020-11-09 09:10:30 [arthas-NettyHttpTelnetBootstrap-4-1] INFO  c.a.a.d.i.n.h.logging.LoggingHandler -[id: 0xe59f20c5, L:/172.20.0.7:3658] READ: [id: 0x3106bb9f, L:/172.20.0.7:3658 - R:/172.20.0.32:55952]
2020-11-09 09:10:30 [arthas-NettyHttpTelnetBootstrap-4-1] INFO  c.a.a.d.i.n.h.logging.LoggingHandler -[id: 0xe59f20c5, L:/172.20.0.7:3658] READ COMPLETE
2020-11-09 09:11:23 [arthas-NettyHttpTelnetBootstrap-4-1] INFO  c.a.a.d.i.n.h.logging.LoggingHandler -[id: 0xe59f20c5, L:/172.20.0.7:3658] READ: [id: 0xa8167699, L:/172.20.0.7:3658 - R:/172.20.0.32:56812]
2020-11-09 09:11:23 [arthas-NettyHttpTelnetBootstrap-4-1] INFO  c.a.a.d.i.n.h.logging.LoggingHandler -[id: 0xe59f20c5, L:/172.20.0.7:3658] READ COMPLETE
2020-11-09 09:11:26 [arthas-NettyHttpTelnetBootstrap-4-1] INFO  c.a.a.d.i.n.h.logging.LoggingHandler -[id: 0xe59f20c5, L:/172.20.0.7:3658] READ: [id: 0x79a99dcb, L:/172.20.0.7:3658 - R:/172.20.0.32:56846]
2020-11-09 09:11:26 [arthas-NettyHttpTelnetBootstrap-4-1] INFO  c.a.a.d.i.n.h.logging.LoggingHandler -[id: 0xe59f20c5, L:/172.20.0.7:3658] READ COMPLETE
2020-11-09 09:13:46 [arthas-NettyHttpTelnetBootstrap-4-1] INFO  c.a.a.d.i.n.h.logging.LoggingHandler -[id: 0xe59f20c5, L:/172.20.0.7:3658] READ: [id: 0xa8c90208, L:/172.20.0.7:3658 - R:/172.20.0.216:50978]
2020-11-09 09:13:46 [arthas-NettyHttpTelnetBootstrap-4-1] INFO  c.a.a.d.i.n.h.logging.LoggingHandler -[id: 0xe59f20c5, L:/172.20.0.7:3658] READ COMPLETE
2020-11-09 09:13:46 [arthas-NettyHttpTelnetBootstrap-4-1] INFO  c.a.a.d.i.n.h.logging.LoggingHandler -[id: 0xe59f20c5, L:/172.20.0.7:3658] READ: [id: 0x204c952d, L:/172.20.0.7:3658 - R:/172.20.0.216:50979]
2020-11-09 09:13:46 [arthas-NettyHttpTelnetBootstrap-4-1] INFO  c.a.a.d.i.n.h.logging.LoggingHandler -[id: 0xe59f20c5, L:/172.20.0.7:3658] READ COMPLETE
2020-11-09 09:13:46 [arthas-NettyHttpTelnetBootstrap-4-1] INFO  c.a.a.d.i.n.h.logging.LoggingHandler -[id: 0xe59f20c5, L:/172.20.0.7:3658] READ: [id: 0x4b3d954f, L:/172.20.0.7:3658 - R:/172.20.0.216:50981]
2020-11-09 09:13:46 [arthas-NettyHttpTelnetBootstrap-4-1] INFO  c.a.a.d.i.n.h.logging.LoggingHandler -[id: 0xe59f20c5, L:/172.20.0.7:3658] READ COMPLETE
2020-11-09 09:13:46 [arthas-NettyHttpTelnetBootstrap-4-1] INFO  c.a.a.d.i.n.h.logging.LoggingHandler -[id: 0xe59f20c5, L:/172.20.0.7:3658] READ: [id: 0xe0d68b61, L:/172.20.0.7:3658 - R:/172.20.0.216:50982]
2020-11-09 09:13:46 [arthas-NettyHttpTelnetBootstrap-4-1] INFO  c.a.a.d.i.n.h.logging.LoggingHandler -[id: 0xe59f20c5, L:/172.20.0.7:3658] READ COMPLETE
2020-11-09 09:13:46 [arthas-NettyHttpTelnetBootstrap-4-1] INFO  c.a.a.d.i.n.h.logging.LoggingHandler -[id: 0xe59f20c5, L:/172.20.0.7:3658] READ: [id: 0x3615dd7b, L:/172.20.0.7:3658 - R:/172.20.0.216:50983]
2020-11-09 09:13:46 [arthas-NettyHttpTelnetBootstrap-4-1] INFO  c.a.a.d.i.n.h.logging.LoggingHandler -[id: 0xe59f20c5, L:/172.20.0.7:3658] READ COMPLETE
2020-11-09 09:13:46 [arthas-NettyHttpTelnetBootstrap-4-1] INFO  c.a.a.d.i.n.h.logging.LoggingHandler -[id: 0xe59f20c5, L:/172.20.0.7:3658] READ: [id: 0x59748075, L:/172.20.0.7:3658 - R:/172.20.0.216:50984]
2020-11-09 09:13:46 [arthas-NettyHttpTelnetBootstrap-4-1] INFO  c.a.a.d.i.n.h.logging.LoggingHandler -[id: 0xe59f20c5, L:/172.20.0.7:3658] READ COMPLETE
2020-11-09 09:13:46 [arthas-NettyHttpTelnetBootstrap-4-1] INFO  c.a.a.d.i.n.h.logging.LoggingHandler -[id: 0xe59f20c5, L:/172.20.0.7:3658] READ: [id: 0x1e7c0f2a, L:/172.20.0.7:3658 - R:/172.20.0.216:50985]
2020-11-09 09:13:46 [arthas-NettyHttpTelnetBootstrap-4-1] INFO  c.a.a.d.i.n.h.logging.LoggingHandler -[id: 0xe59f20c5, L:/172.20.0.7:3658] READ COMPLETE
2020-11-09 09:13:46 [arthas-NettyHttpTelnetBootstrap-4-1] INFO  c.a.a.d.i.n.h.logging.LoggingHandler -[id: 0xe59f20c5, L:/172.20.0.7:3658] READ: [id: 0x1a2e54c1, L:/172.20.0.7:3658 - R:/172.20.0.216:50986]
2020-11-09 09:13:46 [arthas-NettyHttpTelnetBootstrap-4-1] INFO  c.a.a.d.i.n.h.logging.LoggingHandler -[id: 0xe59f20c5, L:/172.20.0.7:3658] READ COMPLETE
2020-11-09 09:13:46 [arthas-NettyHttpTelnetBootstrap-4-1] INFO  c.a.a.d.i.n.h.logging.LoggingHandler -[id: 0xe59f20c5, L:/172.20.0.7:3658] READ: [id: 0xec363484, L:/172.20.0.7:3658 - R:/172.20.0.216:50987]
2020-11-09 09:13:46 [arthas-NettyHttpTelnetBootstrap-4-1] INFO  c.a.a.d.i.n.h.logging.LoggingHandler -[id: 0xe59f20c5, L:/172.20.0.7:3658] READ COMPLETE
2020-11-09 09:13:46 [arthas-NettyHttpTelnetBootstrap-4-1] INFO  c.a.a.d.i.n.h.logging.LoggingHandler -[id: 0xe59f20c5, L:/172.20.0.7:3658] READ: [id: 0x43e0f7df, L:/172.20.0.7:3658 - R:/172.20.0.216:50988]
2020-11-09 09:13:46 [arthas-NettyHttpTelnetBootstrap-4-1] INFO  c.a.a.d.i.n.h.logging.LoggingHandler -[id: 0xe59f20c5, L:/172.20.0.7:3658] READ COMPLETE
2020-11-09 09:13:46 [arthas-NettyHttpTelnetBootstrap-4-1] INFO  c.a.a.d.i.n.h.logging.LoggingHandler -[id: 0xe59f20c5, L:/172.20.0.7:3658] READ: [id: 0x9ff9ecc0, L:/172.20.0.7:3658 - R:/172.20.0.216:50989]
2020-11-09 09:13:46 [arthas-NettyHttpTelnetBootstrap-4-1] INFO  c.a.a.d.i.n.h.logging.LoggingHandler -[id: 0xe59f20c5, L:/172.20.0.7:3658] READ COMPLETE
2020-11-09 09:13:46 [arthas-NettyHttpTelnetBootstrap-4-1] INFO  c.a.a.d.i.n.h.logging.LoggingHandler -[id: 0xe59f20c5, L:/172.20.0.7:3658] READ: [id: 0xea95b688, L:/172.20.0.7:3658 - R:/172.20.0.216:50992]
2020-11-09 09:13:46 [arthas-NettyHttpTelnetBootstrap-4-1] INFO  c.a.a.d.i.n.h.logging.LoggingHandler -[id: 0xe59f20c5, L:/172.20.0.7:3658] READ COMPLETE
2020-11-09 09:13:59 [arthas-NettyHttpTelnetBootstrap-4-1] INFO  c.a.a.d.i.n.h.logging.LoggingHandler -[id: 0xe59f20c5, L:/172.20.0.7:3658] READ: [id: 0xd6ce5867, L:/172.20.0.7:3658 - R:/172.20.0.216:51003]
2020-11-09 09:13:59 [arthas-NettyHttpTelnetBootstrap-4-1] INFO  c.a.a.d.i.n.h.logging.LoggingHandler -[id: 0xe59f20c5, L:/172.20.0.7:3658] READ COMPLETE
2020-11-09 09:13:59 [arthas-NettyHttpTelnetBootstrap-4-2] INFO  c.t.a.core.shell.term.impl.Helper -Loaded arthas keymap file from com/taobao/arthas/core/shell/term/readline/inputrc
2020-11-09 09:14:21 [arthas-NettyWebsocketTtyBootstrap-5-1] INFO  c.a.a.d.i.n.h.logging.LoggingHandler -[id: 0xa953fdd9, L:/172.20.0.7:8563] READ: [id: 0x84c1bfd1, L:/172.20.0.7:8563 - R:/172.20.0.216:51013]
2020-11-09 09:14:21 [arthas-NettyWebsocketTtyBootstrap-5-1] INFO  c.a.a.d.i.n.h.logging.LoggingHandler -[id: 0xa953fdd9, L:/172.20.0.7:8563] READ COMPLETE
2020-11-09 09:14:21 [arthas-NettyWebsocketTtyBootstrap-5-1] INFO  c.a.a.d.i.n.h.logging.LoggingHandler -[id: 0xa953fdd9, L:/172.20.0.7:8563] READ: [id: 0xbaaeb349, L:/172.20.0.7:8563 - R:/172.20.0.216:51014]
2020-11-09 09:14:21 [arthas-NettyWebsocketTtyBootstrap-5-1] INFO  c.a.a.d.i.n.h.logging.LoggingHandler -[id: 0xa953fdd9, L:/172.20.0.7:8563] READ COMPLETE
2020-11-09 09:14:21 [arthas-NettyWebsocketTtyBootstrap-5-1] INFO  c.a.a.d.i.n.h.logging.LoggingHandler -[id: 0xa953fdd9, L:/172.20.0.7:8563] READ: [id: 0xf54746f9, L:/172.20.0.7:8563 - R:/172.20.0.216:51015]
2020-11-09 09:14:21 [arthas-NettyWebsocketTtyBootstrap-5-1] INFO  c.a.a.d.i.n.h.logging.LoggingHandler -[id: 0xa953fdd9, L:/172.20.0.7:8563] READ COMPLETE
2020-11-09 09:14:21 [arthas-NettyWebsocketTtyBootstrap-5-1] INFO  c.a.a.d.i.n.h.logging.LoggingHandler -[id: 0xa953fdd9, L:/172.20.0.7:8563] READ: [id: 0xa9f7503d, L:/172.20.0.7:8563 - R:/172.20.0.216:51016]
2020-11-09 09:14:21 [arthas-NettyWebsocketTtyBootstrap-5-1] INFO  c.a.a.d.i.n.h.logging.LoggingHandler -[id: 0xa953fdd9, L:/172.20.0.7:8563] READ COMPLETE
2020-11-09 09:14:21 [arthas-NettyWebsocketTtyBootstrap-5-1] INFO  c.a.a.d.i.n.h.logging.LoggingHandler -[id: 0xa953fdd9, L:/172.20.0.7:8563] READ: [id: 0x914da900, L:/172.20.0.7:8563 - R:/172.20.0.216:51017]
2020-11-09 09:14:21 [arthas-NettyWebsocketTtyBootstrap-5-1] INFO  c.a.a.d.i.n.h.logging.LoggingHandler -[id: 0xa953fdd9, L:/172.20.0.7:8563] READ COMPLETE
2020-11-09 09:14:21 [arthas-NettyWebsocketTtyBootstrap-5-1] INFO  c.a.a.d.i.n.h.logging.LoggingHandler -[id: 0xa953fdd9, L:/172.20.0.7:8563] READ: [id: 0x32abd270, L:/172.20.0.7:8563 - R:/172.20.0.216:51018]
2020-11-09 09:14:21 [arthas-NettyWebsocketTtyBootstrap-5-1] INFO  c.a.a.d.i.n.h.logging.LoggingHandler -[id: 0xa953fdd9, L:/172.20.0.7:8563] READ COMPLETE
2020-11-09 09:14:21 [arthas-NettyWebsocketTtyBootstrap-5-1] INFO  c.a.a.d.i.n.h.logging.LoggingHandler -[id: 0xa953fdd9, L:/172.20.0.7:8563] READ: [id: 0xc5c4ecf7, L:/172.20.0.7:8563 - R:/172.20.0.216:51019]
2020-11-09 09:14:21 [arthas-NettyWebsocketTtyBootstrap-5-1] INFO  c.a.a.d.i.n.h.logging.LoggingHandler -[id: 0xa953fdd9, L:/172.20.0.7:8563] READ COMPLETE
2020-11-09 09:14:21 [arthas-NettyWebsocketTtyBootstrap-5-1] INFO  c.a.a.d.i.n.h.logging.LoggingHandler -[id: 0xa953fdd9, L:/172.20.0.7:8563] READ: [id: 0xb044fcca, L:/172.20.0.7:8563 - R:/172.20.0.216:51020]
2020-11-09 09:14:21 [arthas-NettyWebsocketTtyBootstrap-5-1] INFO  c.a.a.d.i.n.h.logging.LoggingHandler -[id: 0xa953fdd9, L:/172.20.0.7:8563] READ COMPLETE
2020-11-09 09:14:21 [arthas-NettyWebsocketTtyBootstrap-5-1] INFO  c.a.a.d.i.n.h.logging.LoggingHandler -[id: 0xa953fdd9, L:/172.20.0.7:8563] READ: [id: 0xf2f6cf09, L:/172.20.0.7:8563 - R:/172.20.0.216:51021]
2020-11-09 09:14:21 [arthas-NettyWebsocketTtyBootstrap-5-1] INFO  c.a.a.d.i.n.h.logging.LoggingHandler -[id: 0xa953fdd9, L:/172.20.0.7:8563] READ COMPLETE
2020-11-09 09:14:21 [arthas-NettyWebsocketTtyBootstrap-5-1] INFO  c.a.a.d.i.n.h.logging.LoggingHandler -[id: 0xa953fdd9, L:/172.20.0.7:8563] READ: [id: 0xdf5c083a, L:/172.20.0.7:8563 - R:/172.20.0.216:51022]
2020-11-09 09:14:21 [arthas-NettyWebsocketTtyBootstrap-5-1] INFO  c.a.a.d.i.n.h.logging.LoggingHandler -[id: 0xa953fdd9, L:/172.20.0.7:8563] READ COMPLETE
2020-11-09 09:14:21 [arthas-NettyWebsocketTtyBootstrap-5-1] INFO  c.a.a.d.i.n.h.logging.LoggingHandler -[id: 0xa953fdd9, L:/172.20.0.7:8563] READ: [id: 0x83f4d84f, L:/172.20.0.7:8563 - R:/172.20.0.216:51023]
2020-11-09 09:14:21 [arthas-NettyWebsocketTtyBootstrap-5-1] INFO  c.a.a.d.i.n.h.logging.LoggingHandler -[id: 0xa953fdd9, L:/172.20.0.7:8563] READ COMPLETE
2020-11-09 09:14:21 [arthas-NettyWebsocketTtyBootstrap-5-1] INFO  c.a.a.d.i.n.h.logging.LoggingHandler -[id: 0xa953fdd9, L:/172.20.0.7:8563] READ: [id: 0xe72f0553, L:/172.20.0.7:8563 - R:/172.20.0.216:51025]
2020-11-09 09:14:21 [arthas-NettyWebsocketTtyBootstrap-5-1] INFO  c.a.a.d.i.n.h.logging.LoggingHandler -[id: 0xa953fdd9, L:/172.20.0.7:8563] READ COMPLETE
2020-11-09 09:17:32 [arthas-NettyHttpTelnetBootstrap-4-1] INFO  c.a.a.d.i.n.h.logging.LoggingHandler -[id: 0xe59f20c5, L:/172.20.0.7:3658] READ: [id: 0x289f24ed, L:/172.20.0.7:3658 - R:/172.20.0.32:58014]
2020-11-09 09:17:32 [arthas-NettyHttpTelnetBootstrap-4-1] INFO  c.a.a.d.i.n.h.logging.LoggingHandler -[id: 0xe59f20c5, L:/172.20.0.7:3658] READ COMPLETE
2020-11-09 09:19:05 [arthas-NettyWebsocketTtyBootstrap-5-1] INFO  c.a.a.d.i.n.h.logging.LoggingHandler -[id: 0xa953fdd9, L:/172.20.0.7:8563] READ: [id: 0x8bf8ee02, L:/172.20.0.7:8563 - R:/172.20.0.216:51240]
2020-11-09 09:19:05 [arthas-NettyWebsocketTtyBootstrap-5-1] INFO  c.a.a.d.i.n.h.logging.LoggingHandler -[id: 0xa953fdd9, L:/172.20.0.7:8563] READ COMPLETE
2020-11-09 09:19:05 [arthas-NettyWebsocketTtyBootstrap-5-4] INFO  c.t.a.core.shell.term.impl.Helper -Loaded arthas keymap file from com/taobao/arthas/core/shell/term/readline/inputrc
2020-11-09 09:26:42 [arthas-NettyWebsocketTtyBootstrap-5-1] INFO  c.a.a.d.i.n.h.logging.LoggingHandler -[id: 0xa953fdd9, L:/172.20.0.7:8563] READ: [id: 0x02aa24f7, L:/172.20.0.7:8563 - R:/172.20.0.216:51976]
2020-11-09 09:26:42 [arthas-NettyWebsocketTtyBootstrap-5-1] INFO  c.a.a.d.i.n.h.logging.LoggingHandler -[id: 0xa953fdd9, L:/172.20.0.7:8563] READ COMPLETE
2020-11-09 09:26:42 [arthas-NettyWebsocketTtyBootstrap-5-1] INFO  c.t.a.core.shell.term.impl.Helper -Loaded arthas keymap file from com/taobao/arthas/core/shell/term/readline/inputrc
hengyunabc commented 3 years ago

@Ronaltn 我本地试了下，不能重现。 可以连接到本地的端口，然后用 session 命令查看具体的 agentId。

Ronaltn commented 3 years ago

@Ronaltn 我本地试了下，不能重现。 可以连接到本地的端口，然后用 session 命令查看具体的 agentId。

我刚才试了，直接连接到springboot应用节点用session命令查到了agentId，如下图： 微信截图_20201109172050 再连接到tunnel-server的webconle，输入agentId，连不上。 springboot应用中的arthas日志：

2020-11-09 17:15:21 [arthas-NettyHttpTelnetBootstrap-4-1] INFO  c.a.a.d.i.n.h.logging.LoggingHandler -[id: 0x0655a2b3, L:/172.20.3.251:3658] READ: [id: 0xd5462b70, L:/172.20.3.251:3658 - R:/172.20.0.216:50260]
2020-11-09 17:15:21 [arthas-NettyHttpTelnetBootstrap-4-1] INFO  c.a.a.d.i.n.h.logging.LoggingHandler -[id: 0x0655a2b3, L:/172.20.3.251:3658] READ COMPLETE
2020-11-09 17:15:21 [arthas-NettyHttpTelnetBootstrap-4-1] INFO  c.a.a.d.i.n.h.logging.LoggingHandler -[id: 0x0655a2b3, L:/172.20.3.251:3658] READ: [id: 0x46b5963b, L:/172.20.3.251:3658 - R:/172.20.0.216:50261]
2020-11-09 17:15:21 [arthas-NettyHttpTelnetBootstrap-4-1] INFO  c.a.a.d.i.n.h.logging.LoggingHandler -[id: 0x0655a2b3, L:/172.20.3.251:3658] READ COMPLETE
2020-11-09 17:15:21 [arthas-NettyHttpTelnetBootstrap-4-1] INFO  c.a.a.d.i.n.h.logging.LoggingHandler -[id: 0x0655a2b3, L:/172.20.3.251:3658] READ: [id: 0x3f783acd, L:/172.20.3.251:3658 - R:/172.20.0.216:50264]
2020-11-09 17:15:21 [arthas-NettyHttpTelnetBootstrap-4-1] INFO  c.a.a.d.i.n.h.logging.LoggingHandler -[id: 0x0655a2b3, L:/172.20.3.251:3658] READ COMPLETE
2020-11-09 17:15:21 [arthas-NettyHttpTelnetBootstrap-4-1] INFO  c.a.a.d.i.n.h.logging.LoggingHandler -[id: 0x0655a2b3, L:/172.20.3.251:3658] READ: [id: 0x50778f3d, L:/172.20.3.251:3658 - R:/172.20.0.216:50265]
2020-11-09 17:15:21 [arthas-NettyHttpTelnetBootstrap-4-1] INFO  c.a.a.d.i.n.h.logging.LoggingHandler -[id: 0x0655a2b3, L:/172.20.3.251:3658] READ COMPLETE
2020-11-09 17:15:21 [arthas-NettyHttpTelnetBootstrap-4-1] INFO  c.a.a.d.i.n.h.logging.LoggingHandler -[id: 0x0655a2b3, L:/172.20.3.251:3658] READ: [id: 0x1195cc1b, L:/172.20.3.251:3658 - R:/172.20.0.216:50266]
2020-11-09 17:15:21 [arthas-NettyHttpTelnetBootstrap-4-1] INFO  c.a.a.d.i.n.h.logging.LoggingHandler -[id: 0x0655a2b3, L:/172.20.3.251:3658] READ COMPLETE
2020-11-09 17:15:21 [arthas-NettyHttpTelnetBootstrap-4-1] INFO  c.a.a.d.i.n.h.logging.LoggingHandler -[id: 0x0655a2b3, L:/172.20.3.251:3658] READ: [id: 0xd5ffaf53, L:/172.20.3.251:3658 - R:/172.20.0.216:50268]
2020-11-09 17:15:21 [arthas-NettyHttpTelnetBootstrap-4-1] INFO  c.a.a.d.i.n.h.logging.LoggingHandler -[id: 0x0655a2b3, L:/172.20.3.251:3658] READ COMPLETE
2020-11-09 17:15:21 [arthas-NettyHttpTelnetBootstrap-4-1] INFO  c.a.a.d.i.n.h.logging.LoggingHandler -[id: 0x0655a2b3, L:/172.20.3.251:3658] READ: [id: 0x481dcd11, L:/172.20.3.251:3658 - R:/172.20.0.216:50267]
2020-11-09 17:15:21 [arthas-NettyHttpTelnetBootstrap-4-1] INFO  c.a.a.d.i.n.h.logging.LoggingHandler -[id: 0x0655a2b3, L:/172.20.3.251:3658] READ COMPLETE
2020-11-09 17:15:21 [arthas-NettyHttpTelnetBootstrap-4-1] INFO  c.a.a.d.i.n.h.logging.LoggingHandler -[id: 0x0655a2b3, L:/172.20.3.251:3658] READ: [id: 0xbc3c7f84, L:/172.20.3.251:3658 - R:/172.20.0.216:50269]
2020-11-09 17:15:21 [arthas-NettyHttpTelnetBootstrap-4-1] INFO  c.a.a.d.i.n.h.logging.LoggingHandler -[id: 0x0655a2b3, L:/172.20.3.251:3658] READ COMPLETE
2020-11-09 17:15:21 [arthas-NettyHttpTelnetBootstrap-4-1] INFO  c.a.a.d.i.n.h.logging.LoggingHandler -[id: 0x0655a2b3, L:/172.20.3.251:3658] READ: [id: 0x0855300c, L:/172.20.3.251:3658 - R:/172.20.0.216:50270]
2020-11-09 17:15:21 [arthas-NettyHttpTelnetBootstrap-4-1] INFO  c.a.a.d.i.n.h.logging.LoggingHandler -[id: 0x0655a2b3, L:/172.20.3.251:3658] READ COMPLETE
2020-11-09 17:15:21 [arthas-NettyHttpTelnetBootstrap-4-1] INFO  c.a.a.d.i.n.h.logging.LoggingHandler -[id: 0x0655a2b3, L:/172.20.3.251:3658] READ: [id: 0x903ccf68, L:/172.20.3.251:3658 - R:/172.20.0.216:50272]
2020-11-09 17:15:21 [arthas-NettyHttpTelnetBootstrap-4-1] INFO  c.a.a.d.i.n.h.logging.LoggingHandler -[id: 0x0655a2b3, L:/172.20.3.251:3658] READ COMPLETE
2020-11-09 17:15:21 [arthas-NettyHttpTelnetBootstrap-4-1] INFO  c.a.a.d.i.n.h.logging.LoggingHandler -[id: 0x0655a2b3, L:/172.20.3.251:3658] READ: [id: 0x49a64dd3, L:/172.20.3.251:3658 - R:/172.20.0.216:50273]
2020-11-09 17:15:21 [arthas-NettyHttpTelnetBootstrap-4-1] INFO  c.a.a.d.i.n.h.logging.LoggingHandler -[id: 0x0655a2b3, L:/172.20.3.251:3658] READ COMPLETE
2020-11-09 17:15:21 [arthas-NettyHttpTelnetBootstrap-4-1] INFO  c.a.a.d.i.n.h.logging.LoggingHandler -[id: 0x0655a2b3, L:/172.20.3.251:3658] READ: [id: 0x3f51aa70, L:/172.20.3.251:3658 - R:/172.20.0.216:50277]
2020-11-09 17:15:21 [arthas-NettyHttpTelnetBootstrap-4-1] INFO  c.a.a.d.i.n.h.logging.LoggingHandler -[id: 0x0655a2b3, L:/172.20.3.251:3658] READ COMPLETE
2020-11-09 17:15:25 [arthas-NettyHttpTelnetBootstrap-4-1] INFO  c.a.a.d.i.n.h.logging.LoggingHandler -[id: 0x0655a2b3, L:/172.20.3.251:3658] READ: [id: 0x7a6bb67c, L:/172.20.3.251:3658 - R:/172.20.0.216:50287]
2020-11-09 17:15:25 [arthas-NettyHttpTelnetBootstrap-4-1] INFO  c.a.a.d.i.n.h.logging.LoggingHandler -[id: 0x0655a2b3, L:/172.20.3.251:3658] READ COMPLETE
2020-11-09 17:15:25 [arthas-NettyHttpTelnetBootstrap-4-2] INFO  c.t.a.core.shell.term.impl.Helper -Loaded arthas keymap file from com/taobao/arthas/core/shell/term/readline/inputrc

tunnel-server日志：

2020-11-09 17:16:15.127  INFO 1 --- [Server-boss-1-1] io.netty.handler.logging.LoggingHandler  : [id: 0x4472dca3, L:/0.0.0.0:7777] READ: [id: 0xe8c19318, L:/172.20.0.241:7777 - R:/172.20.0.216:50360]
2020-11-09 17:16:15.127  INFO 1 --- [Server-boss-1-1] io.netty.handler.logging.LoggingHandler  : [id: 0x4472dca3, L:/0.0.0.0:7777] READ COMPLETE
2020-11-09 17:16:15.168  INFO 1 --- [rver-worker-3-2] c.a.a.t.server.TunnelSocketFrameHandler  : websocket handshake complete, uri: /ws?method=connectArthas&id=172.20.3.251
2020-11-09 17:16:15.169  INFO 1 --- [rver-worker-3-2] c.a.a.t.server.TunnelSocketFrameHandler  : try to connect to arthas agent, id: 172.20.3.251
2020-11-09 17:16:15.169  INFO 1 --- [rver-worker-3-2] c.a.a.t.server.TunnelSocketFrameHandler  : random clientConnectionId: HHHBQSAMQQ6ISMHVC4OA
2020-11-09 17:16:15.169  INFO 1 --- [rver-worker-3-2] c.a.a.t.server.TunnelSocketFrameHandler  : startTunnel response: response:/?method=startTunnel&id=172.20.3.251&clientConnectionId=HHHBQSAMQQ6ISMHVC4OA
2020-11-09 17:16:15.169  INFO 1 --- [rver-worker-3-2] c.a.a.t.server.TunnelSocketFrameHandler  : browser connect waitting for arthas agent open tunnel
2020-11-09 17:16:15.171  WARN 1 --- [rver-worker-3-1] io.netty.channel.DefaultChannelPipeline  : An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.

java.io.IOException: Connection reset by peer
    at sun.nio.ch.FileDispatcherImpl.read0(Native Method) ~[na:1.8.0_265]
    at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) ~[na:1.8.0_265]
    at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) ~[na:1.8.0_265]
    at sun.nio.ch.IOUtil.read(IOUtil.java:192) ~[na:1.8.0_265]
    at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:377) ~[na:1.8.0_265]
    at io.netty.buffer.PooledByteBuf.setBytes(PooledByteBuf.java:253) ~[netty-buffer-4.1.46.Final.jar!/:4.1.46.Final]
    at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1133) ~[netty-buffer-4.1.46.Final.jar!/:4.1.46.Final]
    at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:350) ~[netty-transport-4.1.46.Final.jar!/:4.1.46.Final]
    at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:148) ~[netty-transport-4.1.46.Final.jar!/:4.1.46.Final]
    at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) [netty-transport-4.1.46.Final.jar!/:4.1.46.Final]
    at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650) [netty-transport-4.1.46.Final.jar!/:4.1.46.Final]
    at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576) [netty-transport-4.1.46.Final.jar!/:4.1.46.Final]
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [netty-transport-4.1.46.Final.jar!/:4.1.46.Final]
    at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-common-4.1.46.Final.jar!/:4.1.46.Final]
    at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.46.Final.jar!/:4.1.46.Final]
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-common-4.1.46.Final.jar!/:4.1.46.Final]
    at java.lang.Thread.run(Thread.java:748) [na:1.8.0_265]

2020-11-09 17:16:35.170 ERROR 1 --- [rver-worker-3-2] c.a.a.t.server.TunnelSocketFrameHandler  : browser connect wait for arthas agent open tunnel timeout, agentId: [172.20.3.251], clientConnectionId: HHHBQSAMQQ6ISMHVC4OA
hengyunabc commented 3 years ago

@Ronaltn 参考下 tunnel server的工作流程： https://github.com/alibaba/arthas/tree/master/tunnel-server

从日志上来看，agent连接到 tunnel server了，并且也查找到了对应的链接。

然后 tunnel server尝试让 agent发一条新的链接到，然后等超时了。 所以这里可能要检查下 agent的日志，看有没有收到 response:/?method=startTunnel 这样子的消息。

另外，可以在 agent机器上检查， netstat -antp | grep 7777 ，看有多个tcp连接到 tunnel server上。

hengyunabc commented 3 years ago

另外，tunnel server是部了一台还是多台？多台的话可能会出错，因为 agent可能第二条tcp连接到不同的机器上。

cookieyoung commented 3 years ago

应该是由于docker网络会将长时间没有进行数据交互的连接给踢出，导致tunnel server发送时网络已经断开从而报错。我在k8s环境中也遇到这种情况。 虽然tunnel client有重连机制但是没有进行心跳检测所以上述情况会导致tunnel client的连接无法正常关闭，因而无法重连。 建议：tunnle client增加心跳检测

qeddfd commented 3 years ago

应该是由于docker网络会将长时间没有进行数据交互的连接给踢出，导致tunnel server发送时网络已经断开从而报错。我在k8s环境中也遇到这种情况。 虽然tunnel client有重连机制但是没有进行心跳检测所以上述情况会导致tunnel client的连接无法正常关闭，因而无法重连。 建议：tunnle client增加心跳检测

我觉得你这边分析的最贴合，请问我们自己如何在client中增加心跳检查

cookieyoung commented 3 years ago

应该是由于docker网络会将长时间没有进行数据交互的连接给踢出，导致tunnel server发送时网络已经断开从而报错。我在k8s环境中也遇到这种情况。 虽然tunnel client有重连机制但是没有进行心跳检测所以上述情况会导致tunnel client的连接无法正常关闭，因而无法重连。 建议：tunnle client增加心跳检测

我觉得你这边分析的最贴合，请问我们自己如何在client中增加心跳检查

修改了tunnel-client源码，channelPipeline中增加了IdleStateHandler来检测连接是否idle的，如果连接idle就pingwebsocket让网络连接保持不被踢出

hengyunabc commented 3 years ago

修改了tunnel-client源码，channelPipeline中增加了IdleStateHandler来检测连接是否idle的，如果连接idle就pingwebsocket让网络连接保持不被踢出

@cookieyoung 欢迎提交PR😁

logan2013 commented 3 years ago

@hengyunabc 请问什么时候推到仓库中提供maven依赖

hengyunabc commented 3 years ago

@hengyunabc 请问什么时候推到仓库中提供maven依赖

仓库里已经有了： http://search.maven.org/classic/#search%7Cgav%7C1%7Cg%3A%22com.taobao.arthas%22%20AND%20a%3A%22arthas-spring-boot-starter%22

logan2013 commented 3 years ago

@cookieyoung @hengyunabc 更新到了最新版本3.4.8发现没解决，还是同样的问题

hengyunabc commented 3 years ago

@cookieyoung @hengyunabc 更新到了最新版本3.4.8发现没解决，还是同样的问题

检查下 arthas 本身是否用的最新的3.4.8版本。