search

alibaba / nacos

an easy-to-use dynamic service discovery, configuration and service management platform for building cloud native applications.
https://nacos.io
Apache License 2.0
29.46k stars 12.66k forks source link

error: Cannot create stream 545225 greater than Last-Stream-ID 545223 from GOAWAY. #12298

Open ranLee1 opened 1 week ago

ranLee1 commented 1 week ago

Describe the bug Automatic retry registration failed, restart the client program to register Occasionally, the gateway cannot access the subservice until the gateway times out.😢 Server reports an error: 

Cannot create stream 545225 greater than Last-Stream-ID 545223 from GOAWAY.

Client reports an error:

Send request fail, request=ServiceQueryRequest{headers={app=unknown}, requestId='null'}, retryTimes=0,errorMessage=Client not connected,current status:UNHEALTHY
Send request fail, request=ServiceQueryRequest{headers={}, requestId='null'}, retryTimes=0,errorMessage=java.util.concurrent.TimeoutException: Waited 3000 milliseconds (plus 1751 milliseconds, 965480 nanoseconds delay) but future completed as timeout expired

Expected behavior none Actually behavior none How to Reproduce none

Desktop (please complete the following information):

Additional context nacos-service:

2024-06-28 10:06:20,454 WARN Stream Error

io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2Exception$StreamException: Cannot create stream 545225 greater than Last-Stream-ID 545223 from GOAWAY.
    at io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2Exception.streamError(Http2Exception.java:147)
    at io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2Connection$DefaultEndpoint.checkNewStreamAllowed(DefaultHttp2Connection.java:874)
    at io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2Connection$DefaultEndpoint.createStream(DefaultHttp2Connection.java:748)
    at io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2Connection$DefaultEndpoint.createStream(DefaultHttp2Connection.java:668)
    at io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder$FrameReadListener.onHeadersRead(DefaultHttp2ConnectionDecoder.java:330)
    at io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2InboundFrameLogger$1.onHeadersRead(Http2InboundFrameLogger.java:65)
    at io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2FrameReader$1.processFragment(DefaultHttp2FrameReader.java:457)
    at io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2FrameReader.readHeadersFrame(DefaultHttp2FrameReader.java:464)
    at io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2FrameReader.processPayloadState(DefaultHttp2FrameReader.java:254)
    at io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2FrameReader.readFrame(DefaultHttp2FrameReader.java:160)
    at io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2InboundFrameLogger.readFrame(Http2InboundFrameLogger.java:41)
    at io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder.decodeFrame(DefaultHttp2ConnectionDecoder.java:174)
    at io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2ConnectionHandler$FrameDecoder.decode(Http2ConnectionHandler.java:378)
    at io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2ConnectionHandler.decode(Http2ConnectionHandler.java:438)
    at io.grpc.netty.shaded.io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:505)
    at io.grpc.netty.shaded.io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:444)
    at io.grpc.netty.shaded.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:283)
    at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
    at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
    at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
    at io.grpc.netty.shaded.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1421)
    at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
    at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
    at io.grpc.netty.shaded.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930)
    at io.grpc.netty.shaded.io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:794)
    at io.grpc.netty.shaded.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:424)
    at io.grpc.netty.shaded.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:326)
    at io.grpc.netty.shaded.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:918)
    at io.grpc.netty.shaded.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    at io.grpc.netty.shaded.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    at java.lang.Thread.run(Thread.java:748)

nacos-client:

2024-06-28 10:06:31.360  ERROR [com.alibaba.nacos.client.naming.updater.0] com.alibaba.nacos.common.utils.LoggerUtils: Send request fail, request=ServiceQueryRequest{headers={app=unknown}, requestId='null'}, retryTimes=0,errorMessage=Client not connected,current status:UNHEALTHY
2024-06-28 10:06:32.214  ERROR [com.alibaba.nacos.client.naming.updater.0] com.alibaba.nacos.common.utils.LoggerUtils: Send request fail, request=ServiceQueryRequest{headers={app=unknown}, requestId='null'}, retryTimes=1,errorMessage=Client not connected,current status:UNHEALTHY
2024-06-28 10:06:32.214  ERROR [com.alibaba.nacos.client.naming.updater.1] com.alibaba.nacos.common.utils.LoggerUtils: Send request fail, request=ServiceQueryRequest{headers={app=unknown}, requestId='null'}, retryTimes=0,errorMessage=Client not connected,current status:UNHEALTHY
2024-06-28 10:06:33.212  ERROR [com.alibaba.nacos.client.naming.updater.1] com.alibaba.nacos.common.utils.LoggerUtils: Send request fail, request=ServiceQueryRequest{headers={app=unknown}, requestId='null'}, retryTimes=1,errorMessage=Client not connected,current status:UNHEALTHY
2024-06-28 10:06:33.212  ERROR [com.alibaba.nacos.client.naming.updater.0] com.alibaba.nacos.common.utils.LoggerUtils: Send request fail, request=ServiceQueryRequest{headers={app=unknown}, requestId='null'}, retryTimes=2,errorMessage=Client not connected,current status:UNHEALTHY
2024-06-28 10:06:53.154  ERROR [com.alibaba.nacos.client.naming.updater.1] com.alibaba.nacos.common.utils.LoggerUtils: Send request fail, request=ServiceQueryRequest{headers={}, requestId='null'}, retryTimes=0,errorMessage=java.util.concurrent.TimeoutException: Waited 3000 milliseconds (plus 1751 milliseconds, 965480 nanoseconds delay) but future completed as timeout expired
2024-06-28 10:06:34.043  ERROR [com.alibaba.nacos.client.naming.updater.1] com.alibaba.nacos.common.utils.LoggerUtils: Send request fail, request=ServiceQueryRequest{headers={app=unknown}, requestId='null'}, retryTimes=2,errorMessage=Client not connected,current status:UNHEALTHY
2024-06-28 10:06:34.043  ERROR [com.alibaba.nacos.client.naming.updater.0] com.alibaba.nacos.common.utils.LoggerUtils: Send request fail, request=ServiceQueryRequest{headers={app=unknown}, requestId='null'}, retryTimes=0,errorMessage=Client not connected,current status:UNHEALTHY
2024-06-28 10:07:42.720  ERROR [com.alibaba.nacos.client.naming.updater.1] com.alibaba.nacos.common.utils.LoggerUtils: Send request fail, request=ServiceQueryRequest{headers={app=unknown}, requestId='null'}, retryTimes=0,errorMessage=Client not connected,current status:UNHEALTHY
2024-06-28 10:08:50.275  ERROR [com.alibaba.nacos.naming.failover] com.alibaba.nacos.client.naming.backups.FailoverReactor$SwitchRefresher: [NA] failed to read failover switch.
 java.lang.OutOfMemoryError: GC overhead limit exceeded
2024-06-28 10:08:16.558  ERROR [com.alibaba.nacos.client.naming.updater.1] com.alibaba.nacos.common.utils.LoggerUtils: Send request fail, request=ServiceQueryRequest{headers={app=unknown}, requestId='null'}, retryTimes=0,errorMessage=Client not connected,current status:UNHEALTHY
2024-06-28 10:08:41.950  ERROR [com.alibaba.nacos.client.naming.updater.0] com.alibaba.nacos.common.utils.LoggerUtils: Send request fail, request=ServiceQueryRequest{headers={app=unknown}, requestId='null'}, retryTimes=0,errorMessage=Client not connected,current status:UNHEALTHY
2024-06-28 10:08:19.358  ERROR [http-nio-6803-exec-33] org.apache.juli.logging.DirectJDKLog: Failed to complete processing of a request
 java.lang.OutOfMemoryError: GC overhead limit exceeded
2024-06-28 10:07:26.297  ERROR [com.alibaba.nacos.client.naming.updater.0] com.alibaba.nacos.common.utils.LoggerUtils: Send request fail, request=ServiceQueryRequest{headers={}, requestId='null'}, retryTimes=0,errorMessage=java.util.concurrent.TimeoutException: Waited 3000 milliseconds (plus 478 milliseconds, 878195 nanoseconds delay) but future completed as timeout expired
KomachiSion commented 5 days ago

Can you upgrade to new nacos server version and new client version and retry?

ranLee1 commented 4 days ago

Thank you very much for your reply. I am currently using SpringCloud Alibaba 2.2.7.RELEASE. What version should I upgrade to for nacos-server and nacos-client