search

apache / dubbo

The java implementation of Apache Dubbo. An RPC and microservice framework.
https://dubbo.apache.org/
Apache License 2.0
40.42k stars 26.41k forks source link

After the provider application is shut down, consumers will still send heartbeat requests to this offline process #13400

Closed songxiaosheng closed 9 months ago

songxiaosheng commented 10 months ago

Environment

Steps to reproduce this issue

  1. Launch Provider Application One
  2. Launch Provider Application Two
  3. Launch Consumer Application One
  4. Close a provider process
  5. Consumer's heartbeat request is abnormal

Pls. provide [GitHub address] to reproduce this issue.

Expected Behavior

After the provider is offline, consumers should not send heartbeat requests to this provider again

Actual Behavior

After the provider application is shut down, consumers will still send heartbeat requests to this offline process

If there is an exception, please attach the exception trace:

[21/11/23 15:25:22:255 CST] dubbo-client-idleCheck-thread-1  INFO header.ReconnectTimerTask:  [DUBBO] Initial connection to HeaderExchangeClient [channel=org.apache.dubbo.remoting.transport.netty4.NettyClient [/172.20.90.220:62262 -> /172.20.90.220:20881]], dubbo version: 3.1.0, current host: 172.20.90.220
[21/11/23 15:25:22:255 CST] dubbo-client-idleCheck-thread-1  INFO netty4.NettyChannel:  [DUBBO] Close netty channel [id: 0xdba1d488, L:/172.20.90.220:62262 ! R:/172.20.90.220:20881], dubbo version: 3.1.0, current host: 172.20.90.220
[21/11/23 15:25:22:260 CST] dubbo-client-idleCheck-thread-1 ERROR netty4.NettyClient:  [DUBBO] Failed to connect to provider server by other reason., dubbo version: 3.1.0, current host: 172.20.90.220, error code: 6-1. This may be caused by network disconnected, go to https://dubbo.apache.org/faq/6/1 to find instructions. 
io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /172.20.90.220:20881
Caused by: java.net.ConnectException: Connection refused
    at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
    at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:715)
    at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:337)
    at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334)
    at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:776)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
    at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
    at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    at java.lang.Thread.run(Thread.java:750)
[21/11/23 15:25:22:262 CST] dubbo-client-idleCheck-thread-1 ERROR header.ReconnectTimerTask:  [DUBBO] Fail to connect to HeaderExchangeClient [channel=org.apache.dubbo.remoting.transport.netty4.NettyClient [/172.20.90.220:62262 -> /172.20.90.220:20881]], dubbo version: 3.1.0, current host: 172.20.90.220
org.apache.dubbo.remoting.RemotingException: client(url: dubbo://172.20.90.220:20881/link.elastic.dubbo.entity.DemoService?anyhost=true&application=dubbo-demo-api-consumer&background=false&category=providers,configurators,routers&check=false&codec=dubbo&deprecated=false&dubbo=2.0.2&dynamic=true&enable-empty-protection=false&file.cache=false&generic=false&heartbeat=60000&interface=link.elastic.dubbo.entity.DemoService&metadata-type=remote&methods=sayHello,sayHello1,sayHelloAsync,sayHello2&pid=61116&qos.enable=false&qos.port=-1&release=3.1.0&service-name-mapping=true&side=consumer&sticky=false) failed to connect to server /172.20.90.220:20881, error message is:Connection refused: /172.20.90.220:20881
    at org.apache.dubbo.remoting.transport.netty4.NettyClient.doConnect(NettyClient.java:195)
    at org.apache.dubbo.remoting.transport.AbstractClient.connect(AbstractClient.java:218)
    at org.apache.dubbo.remoting.transport.AbstractClient.reconnect(AbstractClient.java:272)
    at org.apache.dubbo.remoting.exchange.support.header.HeaderExchangeClient.reconnect(HeaderExchangeClient.java:171)
    at org.apache.dubbo.remoting.exchange.support.header.ReconnectTimerTask.doTask(ReconnectTimerTask.java:49)
    at org.apache.dubbo.remoting.exchange.support.header.AbstractTimerTask.run(AbstractTimerTask.java:87)
    at org.apache.dubbo.common.timer.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:651)
    at org.apache.dubbo.common.timer.HashedWheelTimer$HashedWheelBucket.expireTimeouts(HashedWheelTimer.java:730)
    at org.apache.dubbo.common.timer.HashedWheelTimer$Worker.run(HashedWheelTimer.java:452)
    at java.lang.Thread.run(Thread.java:750)
Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /172.20.90.220:20881
Caused by: java.net.ConnectException: Connection refused
    at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
    at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:715)
    at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:337)
    at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334)
    at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:776)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
    at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
    at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    at java.lang.Thread.run(Thread.java:750)
AlbumenJ commented 10 months ago

Maybe we have fixed it in the latest version

songxiaosheng commented 10 months ago

Maybe we have fixed it in the latest version

There are no problems in the 3.2.8 version test for the time being. I will continue to test the 3.1.0 version to see where it is fixed.

CrazyHZM commented 9 months ago

@songxiaosheng We no longer do some bugfixes, and we only fix security issues in 3.1.x.

Jalyn-X commented 9 months ago

Maybe we have fixed it in the latest version

When will this issue be fixed in which version?

iwangjie commented 9 months ago

Excuse me, how do I fix in 3.1.11, do I have a patch that I can apply?

songxiaosheng commented 2 months ago

Excuse me, how do I fix in 3.1.11, do I have a patch that I can apply? @iwangjie i found maybe it https://github.com/apache/dubbo/commit/7facd5be19ea80ffee05172931b589ddc90ace8a

image

iwangjie commented 2 months ago

Excuse me, how do I fix in 3.1.11, do I have a patch that I can apply? @iwangjie i found maybe it https://github.com/apache/dubbo/commit/7facd5be19ea80ffee05172931b589ddc90ace8a

image

thanks