search

etcd-io / etcd

Distributed reliable key-value store for the most critical data of a distributed system
https://etcd.io
Apache License 2.0
47.95k stars 9.78k forks source link

fetching snapshot from learner rather than leader while new or outdated member joins cluster #17161

Open armstrongli opened 11 months ago

armstrongli commented 11 months ago

What would you like to be added?

let the member fetch latest data from other members(learner > member > leader) to avoid impact to leader to catch the baseline before catching up HEAD Write Ahead Log(WAL) from leader.

Why is this needed?

snapshot is a heavy read action for etcd cluster because of the features of storage backend(boltdb). long read transaction results issues for leader to handle write writes, also data fermentation(COW implementation on boltdb). especially in the cases: a. a new member joins the cluster. b. an outdated member joins back from issues(e.g. network isolation/broken).

with this feature, it can mitigate the issue a lot.

ahrtr commented 11 months ago

This is a useful & valid feature. Any members, which are in sync with the leader, should can be delegated to send a snapshot to another follower. It's especially useful in cross-region cluster, we can delegate to a nearest member based on the locality, so as to avoid across region data transportation.

See also https://www.cockroachlabs.com/docs/stable/architecture/replication-layer#snapshots