search

rpcxio / rpcx-etcd

etcd plugin for rpcx
Apache License 2.0
29 stars 24 forks source link

etcd节点异常离线后又上线，大概率出现服务put与keepalive的租约永久不一致，即etcd永久存在两个租约 #27

Closed xlkness closed 1 year ago

xlkness commented 1 year ago

代码版本：

V0.2.0

代码位置：

store/etcdv3/etcdv3.go:187行，以及store/etcdv3/etcdv3.go:108行，均是调用init逻辑

问题描述：

两处代码位置同时检测到error，108快于187处，于是keepalive逻辑先生成租约1做保活，Put逻辑后生成租约2替换了租约1，但keepalive逻辑已经陷入无法重新读取租约2

日志现场：

（我是单独拷贝的serverplugin/etcdv3.go与store/etcdv3/etcdv3.go修改了函数入参，init与grant加了个整型参，区分不同调用位置）

{"level":"warn","ts":"2023-04-04T09:44:12.433+0800","logger":"etcd-client","caller":"v3@v3.5.1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000281c00/192.168.1.22:2415","attempt":95,"error":"rpc error: code = Canceled desc = grpc: the client connection is closing"}
{"level":"warn","ts":"2023-04-04T09:44:12.459+0800","logger":"etcd-client","caller":"v3@v3.5.1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000281c00/192.168.1.22:2415","attempt":96,"error":"rpc error: code = Canceled desc = grpc: the client connection is closing"}
{"level":"warn","ts":"2023-04-04T09:44:12.483+0800","logger":"etcd-client","caller":"v3@v3.5.1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000281c00/192.168.1.22:2415","attempt":97,"error":"rpc error: code = Canceled desc = grpc: the client connection is closing"}
{"level":"warn","ts":"2023-04-04T09:44:12.508+0800","logger":"etcd-client","caller":"v3@v3.5.1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000281c00/192.168.1.22:2415","attempt":98,"error":"rpc error: code = Canceled desc = grpc: the client connection is closing"}
{"level":"warn","ts":"2023-04-04T09:44:12.534+0800","logger":"etcd-client","caller":"v3@v3.5.1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000281c00/192.168.1.22:2415","attempt":99,"error":"rpc error: code = Canceled desc = grpc: the client connection is closing"}
[2023-04-04 09:44:12.534924752 +0800 CST m=+39.822361058]receive close
[2023-04-04 09:44:12.53495451 +0800 CST m=+39.822390819]entry init:2
[2023-04-04 09:44:12.535125456 +0800 CST m=+39.822561764]entry init:3
[2023-04-04 09:44:12.56585232 +0800 CST m=+39.853288633]entry grant:1, new lease:7587869698810587399
[2023-04-04 09:44:12.56590365 +0800 CST m=+39.853339961]entry keepalive, lease id:7587869698810587399
[2023-04-04 09:44:12.566611356 +0800 CST m=+39.854047666]entry grant:3, new lease:7587869698810587401

keepalive的channel先收到关闭事件，完成租约的创建，Put后创建。

smallnest commented 1 year ago

try v0.3.1 please.