etcd-io / etcd

Distributed reliable key-value store for the most critical data of a distributed system
https://etcd.io
Apache License 2.0
47.82k stars 9.77k forks source link

etcd leader nodes switch frequently #16050

Open im-jinxinwang opened 1 year ago

im-jinxinwang commented 1 year ago

Bug report criteria

What happened?

leader nodes switch frequently, error logs: "ignored out-of-date read index response; local node read indexes queueing up and waiting to be in sync with leader"

What did you expect to happen?

I don't think the problem of frequent master node switching for etcd should occur

How can we reproduce it (as minimally and precisely as possible)?

I can't provide an environment to reproduce this issue, but I can provide log information on three points etcd-4(1).log etcd-5(1).log etcd-6(1).log

Anything else we need to know?

metrics information: etcd-metrics(1).txt

fio command test drive. fsync/fdatasync/sync_file_range in 99.00th=[ 1586]

Etcd version (please run commands below)

```console $ etcd --version etcd Version: 3.5.4 $ etcdctl version # paste output here ```

Etcd configuration (command line flags or environment variables)

# paste your configuration here

Etcd debug information (please run commands below, feel free to obfuscate the IP address or FQDN in the output)

```console $ etcdctl member list -w table # paste output here $ etcdctl --endpoints= endpoint status -w table # paste output here ```

Relevant log output

No response

jmhbnz commented 1 year ago

Hey @WVenus - Can you please confirm what backing storage your etcd instance is using? Also if possible do you have a graph for the etcd_disk_wal_fsync_duration_seconds_bucket metric over time?

im-jinxinwang commented 1 year ago

@jmhbnz Now using SSD as etcd storage. No metric over time, is the current point in time OK? Because the monitoring is not yet connected.

stale[bot] commented 1 year ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 21 days if no further activity occurs. Thank you for your contributions.