Open haojinming opened 1 year ago
If kill etcd leader with SIGKILL(9)
, the following error log will print and NewSession
can continue acting after new leader is elected.
{"level":"warn","ts":"2022-10-27T10:55:32.940+0800","logger":"etcd-client","caller":"v3@v3.5.5/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000257340/172.16.5.32:2379","attempt":0,"error":"rpc error: code = Unavailable desc = etcdserver: request timed out"}
I think it should has same behaviour when kill with SIGSTOP
, isn't it?
@ahrtr How about add a configuration item that controls the timeout of grpc function calls when create the etcd client?
I might be completely wrong here, but I think the cause of this behavior is this option: defaultWaitForReady = grpc.WaitForReady(true)
in /go.etcd.io/etcd/client/v3@v3.5.5/options.go
;
since is not accessible I have tried to set it to false
by temporarily modify the file and in my case it does not hang anymore.
Thanks, I changed grpc.WaitForReady
to false
, it works. But I don't understand why grpc.WaitForReady
is hardcoded to true.
I have no idea, I have found an old issue that proposed to expose some of those options, but it went on stale.
Here is the issue: https://github.com/etcd-io/etcd/issues/13344
SIGSTOP(19) of the etcd leader, new leader won't be selected ?
SIGSTOP(19) of the etcd leader, new leader won't be selected ?
Yes, the main
program hang without any log and action.
Thanks, I changed
grpc.WaitForReady
tofalse
, it works. But I don't understand whygrpc.WaitForReady
is hardcoded to true. @zeminzhou I also encountered the same problem, How to change this value in our code?
I also encountered the same problem, How to change this value in our code?
@huangjiao-heart unfortunately you can't as it is a private variable, you have to change it from the package.
To be more clear, you have to literally open the file where the variable is hardcoded and change it manually. Or either make a fork and change it from there.
I'm still quite new to this, but adding a timeout to the LeaseGrant request seem to raise the appropriate error when the servers are killed. Would there be any negative repercussions if we do so?
There are two cases:
If the ETCD server has been paused by SIGSTOP, the client will wait for the connection ready because of WaitForReady
. It can be solved by exporting option.
If the ETCD server is paused after connection ready, the client will wait for http2 response forever. It should be handled by ctx option when you new session.
I'm not sure that reason about pausing the process. It's freezing. It will impact all the ready connections.
If you want to stop the server, you should use SIGTERM
or SIGKILL
.
If you want to stop the server, you should use SIGTERM or SIGKILL.
This bug can be replicated if the server is stopped by SIGTERM
or SIGKILL
as well.
The freeze occurs during the creation of a new session, where client attempts to send a LeaseGrant request to the server. Because the grpc option WaitForReady
is set to true
, the creation of the RPC is queued until the connection is ready, without returning any errors.
I understand from the provided documentation that this option is set to minimise error responses from transient failures. In that case, I'd like to propose setting a timeout to the LeaseGrant gRPC request specifically during the creation of new sessions. I'm thinking of exposing the timeout duration as a configurable value (I'm thinking of naming it as SessionCreationTimeout
for now) in the Config object in etcd/client/v3/config.go
. Do let me know if this has any negative side effects, otherwise I'll be preparing a PR for this over the next few days :+1:
@AngstyDuck Yes. gRPC has a background goroutine picker to collect available connection to new transporter. I think it's good option to user if they want to disable waitForReady.
What happened?
concurrency.NewSession
hang after etcd server is kill by SIGSTOP(19)What did you expect to happen?
NewSession
can return error after server is killed.How can we reproduce it (as minimally and precisely as possible)?
main
with following codes.pidof etcd leader
Anything else we need to know?
If re-create etcd client after
kill -19
, it can return error. However, in our application, the client is created at the beginning and stored to use in the while lifecycle of the application.Etcd version (please run commands below)
Etcd configuration (command line flags or environment variables)
Etcd debug information (please run commands blow, feel free to obfuscate the IP address or FQDN in the output)
Relevant log output
No response