nacos集群，token只能登录其中两台，另外一台报token invalid!

[parrotiger]

1. nacos 2.1.0 版本
2. 其中一台cpu突然飙升100%，要重启，一直卡住，然后Kill -9 强制停止，重启后，这台能获取集群数据，但是登录报Invalid token
3. 我的是nginx负载3台nacos,轮询策略： 登录成功后，一旦轮询到 这台就报
   2023-07-21 11:51:37,106 DEBUG access denied, request: GET /nacos/v1/cs/configs, reason: token invalid!

[parrotiger]

突然飙升100% ，突然起了很多线程，这个是什么问题

[YunWZ]

token无效, 可能是重启之后,没有正确加载secret

[parrotiger]

要怎么处理才能让另外一台 访问正常

[parrotiger]

我尝试 将有问题的一台 清空 data目录status目录，然后重启，能加入集群，但是 依然是 2023-07-21 11:51:37,106 DEBUG access denied, request: GET /nacos/v1/cs/configs, reason: token invalid!， 配置文件是跟其他两台秘钥是一样的，

如何才能重新加载secret

[YunWZ]

要怎么处理才能让另外一台 访问正常

你说的“登陆”是指登陆控制台么？ 之前控制台认证有个bug，在nginx代理下会有这个问题。解决办法是直接访问某一台服务器的控制台，或者升级到2.2版本

[parrotiger]

发现启动有报错nacos.log,请问是我的nacos哪里出了问题 java.util.concurrent.RejectedExecutionException: Task io.grpc.internal.SerializingExecutor@17cc633c rejected from java.util.concurrent.ThreadPoolExecutor@51c3f053[Running, pool size = 64, a ctive threads = 2, queued tasks = 16384, completed tasks = 159219] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at io.grpc.internal.SerializingExecutor.schedule(SerializingExecutor.java:93) at io.grpc.internal.SerializingExecutor.execute(SerializingExecutor.java:86) at io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener.messagesAvailable(ServerImpl.java:796) at io.grpc.internal.AbstractStream$TransportState.messagesAvailable(AbstractStream.java:165) at io.grpc.internal.MessageDeframer.processBody(MessageDeframer.java:408) at io.grpc.internal.MessageDeframer.deliver(MessageDeframer.java:271) at io.grpc.internal.MessageDeframer.deframe(MessageDeframer.java:177) at io.grpc.internal.AbstractStream$TransportState.deframe(AbstractStream.java:193) at io.grpc.internal.AbstractServerStream$TransportState.inboundDataReceived(AbstractServerStream.java:266) at io.grpc.netty.shaded.io.grpc.netty.NettyServerStream$TransportState.inboundDataReceived(NettyServerStream.java:252) at io.grpc.netty.shaded.io.grpc.netty.NettyServerHandler.onDataRead(NettyServerHandler.java:482) at io.grpc.netty.shaded.io.grpc.netty.NettyServerHandler.access$800(NettyServerHandler.java:101) at io.grpc.netty.shaded.io.grpc.netty.NettyServerHandler$FrameListener.onDataRead(NettyServerHandler.java:791) at io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder$FrameReadListener.onDataRead(DefaultHttp2ConnectionDecoder.java:292) at io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2InboundFrameLogger$1.onDataRead(Http2InboundFrameLogger.java:48) at io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2FrameReader.readDataFrame(DefaultHttp2FrameReader.java:422) at io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2FrameReader.processPayloadState(DefaultHttp2FrameReader.java:251) at io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2FrameReader.readFrame(DefaultHttp2FrameReader.java:160) at io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2InboundFrameLogger.readFrame(Http2InboundFrameLogger.java:41) at io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder.decodeFrame(DefaultHttp2ConnectionDecoder.java:174) at io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2ConnectionHandler$FrameDecoder.decode(Http2ConnectionHandler.java:378) at io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2ConnectionHandler.decode(Http2ConnectionHandler.java:438) at io.grpc.netty.shaded.io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:505) at io.grpc.netty.shaded.io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:444) at io.grpc.netty.shaded.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:283) at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) at io.grpc.netty.shaded.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1422) at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) at io.grpc.netty.shaded.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:931) at io.grpc.netty.shaded.io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.grpc.netty.shaded.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:483) at io.grpc.netty.shaded.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:383) at io.grpc.netty.shaded.io.netty.util.concurrent.SingleThreadEventExecutor$6.run(SingleThreadEventExecutor.java:1044) at io.grpc.netty.shaded.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.grpc.netty.shaded.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2023-07-22 08:47:00,871 WARN Exception processing message

io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2Exception$StreamException: Stream closed before write could take place at io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2Exception.streamError(Http2Exception.java:167) at io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2RemoteFlowController$FlowState.cancel(DefaultHttp2RemoteFlowController.java:481) at io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2RemoteFlowController$1.onStreamClosed(DefaultHttp2RemoteFlowController.java:105) at io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2Connection.notifyClosed(DefaultHttp2Connection.java:356) at io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2Connection$ActiveStreams.removeFromActiveStreams(DefaultHttp2Connection.java:1000) at io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2Connection$ActiveStreams.deactivate(DefaultHttp2Connection.java:956) at io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2Connection$DefaultStream.close(DefaultHttp2Connection.java:512) at io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2Connection$DefaultStream.close(DefaultHttp2Connection.java:518) at io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2ConnectionHandler.closeStream(Http2ConnectionHandler.java:613) at io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder$FrameReadListener.onRstStreamRead(DefaultHttp2ConnectionDecoder.java:408) at io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2InboundFrameLogger$1.onRstStreamRead(Http2InboundFrameLogger.java:80) at io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2FrameReader.readRstStreamFrame(DefaultHttp2FrameReader.java:516) at io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2FrameReader.processPayloadState(DefaultHttp2FrameReader.java:260) at io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2FrameReader.readFrame(DefaultHttp2FrameReader.java:160) at io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2InboundFrameLogger.readFrame(Http2InboundFrameLogger.java:41) at io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder.decodeFrame(DefaultHttp2ConnectionDecoder.java:174) at io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2ConnectionHandler$FrameDecoder.decode(Http2ConnectionHandler.java:378) at io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2ConnectionHandler.decode(Http2ConnectionHandler.java:438) at io.grpc.netty.shaded.io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:505) at io.grpc.netty.shaded.io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:444) at io.grpc.netty.shaded.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:283) at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) at io.grpc.netty.shaded.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1422) at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) at io.grpc.netty.shaded.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:931) at io.grpc.netty.shaded.io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.grpc.netty.shaded.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:483)

[parrotiger]

要怎么处理才能让另外一台 访问正常

你说的“登陆”是指登陆控制台么？ 之前控制台认证有个bug，在nginx代理下会有这个问题。解决办法是直接访问某一台服务器的控制台，或者升级到2.2版本

是的 nginx代理了3台，但是在出问题之前 ，一次登录，nginx轮询3台 nacos一直都没有报 token问题的

[parrotiger]

刚刚清空日志，再次重启，nacos.log显示正常没有错误，但是 访问页面来回刷新还是有这个问题

[parrotiger]

[nacos@iZwz93aodfp826woyujfmkZ logs]$ grep Exception * nacos.log:2023-07-22 09:02:31,566 INFO Creating filter chain: any request, [org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter@cb03411, org.springframework.security.web.context.SecurityContextPersistenceFilter@4aaae508, org.springframework.security.web.header.HeaderWriterFilter@164a62bf, org.springframework.security.web.csrf.CsrfFilter@1640c151, org.springframework.security.web.authentication.logout.LogoutFilter@664e5dee, org.springframework.security.web.savedrequest.RequestCacheAwareFilter@7bc6d27a, org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter@6ab4ba9f, org.springframework.security.web.authentication.AnonymousAuthenticationFilter@4c59e45e, org.springframework.security.web.session.SessionManagementFilter@aaee2a2, org.springframework.security.web.access.ExceptionTranslationFilter@2a32fb6] naming-server.log:java.io.IOException: failed to req API:http://172.xx.xx.94:8848/nacos/v1/ns/operator/cluster/state. code:500 msg: caused: unable to find local peer: 172.xx.xx.94:8848, all peers: []; naming-server.log:java.io.IOException: failed to req API:http://172.xx.xx.82:8848/nacos/v1/ns/operator/cluster/state. code:500 msg: caused: unable to find local peer: 172.xx.xx.82:8848, all peers: []; protocol-distro.log:com.alibaba.nacos.core.distributed.distro.exception.DistroException: [DISTRO-EXCEPTION][DISTRO-FAILED] Get distro snapshot failed! protocol-distro.log:Caused by: com.alibaba.nacos.api.exception.NacosException: No rpc client related to member: Member{ip='172.xx.xx.82', port=8848, state=UP, extendInfo={raftPort=7848, readyToUpgrade=true}} protocol-distro.log:com.alibaba.nacos.core.distributed.distro.exception.DistroException: [DISTRO-EXCEPTION][DISTRO-FAILED] Get distro snapshot failed! protocol-distro.log:Caused by: com.alibaba.nacos.api.exception.NacosException: No rpc client related to member: Member{ip='172.xx.xx.94', port=8848, state=UP, extendInfo={raftPort=7848, readyToUpgrade=true}} [nacos@iZwz93aodfp826woyujfmkZ logs]$

最新找到的错误日志

[parrotiger]

我的应用连接nacos集群也报错： token invalid ,必须停止 有问题的那台才能正常连接，请问如何让nacos集群恢复正常，拜托了

[parrotiger]

是jwt 在集群中不同节点 过期问题吗？

[KomachiSion]

看报错像是重启后的节点互相访问不通啊， 看下cluster.conf的配置和节点本身的ip是否是一致的。

[KomachiSion]

有其他反馈信息吗？

[KomachiSion]

No more response from author, I think this is a env problem.