temeo
commented
10 years ago Outline of current design:
EVS monitors response time for each other node. If the response time exceeds evs.delayed_period, node is added to list of delayed nodes.
The node on list of delayed nodes has associated state (OK or DELAYED, initialized to DELAYED) and counter (0...255, initialzed to 0). Each time check for delayed nodes is done (once in evs.inactive_check_period) and state change is detected counter is incremented. Once counter reaches over 1, node UUID and counter is reported to other nodes.
If node state stays OK over evs.delayed_decay_period, its counter is decremented by one. Once the counter reaches zero, node is removed from delayed list.
Delayed list can be monitored via wsrep_evs_delayed status variable. List is in form of comma separated values, with single element being of format
0aac8e2a-0379-11e4-ba9c-afbf6afe14bc:tcp://192.168.17.13:10031:1The first part is node UUID. Second part starting from tcp:// is low level connectivity endpoint (if known) and the final number is the value of state counter.
Node can be manually evicted from the cluster by assigning node UUID into evs.evict wsrep provider option:
set global wsrep_provider_options='evs.evict=0aac8e2a-0379-11e4-ba9c-afbf6afe14bc';This will add node UUID to evicted list and will trigger group membership change.
At the moment it is possible to evict manually only one node at the time. This will cause problems if several nodes reside behind bad link (like in multi data center case) since membership protocol runs poorly over bad network. Parameter evs.evict parsing should be enhanced to allow several UUIDs at once.
Setting parameter evs.auto_evict to value greater than zero will enable automatic eviction. The operation is roughly the following:
Every node listens to messages reporting delayed nodes. When such a message is received, it is stored for evs.decay_period time.
Messages are iterated over and the following counters are updated
- How many messages have a node reported with counter value over
evs.auto_evict - How many messages have a node reported Messages containing processing node's UUID are ignored.
If at least one candidate is found, all nodes that have been reported by majority of the group are evicted automatically.
Also this approach has a problem if there are several nodes behind bad link. Some kind of heuristics should be applied to either try to keep group intact or to evict all delayed candidates without loosing majority of the group.
This kind of design emerged because:
- There must be some way to gather statistics to make eviction decision, and doing it along with inactive check was the easies choise.
- Simply having OK and DELAYED states for nodes to be evicted would not be enough, since eviction would happen with each partition and this would defeat the purpose of partition tolerant algorithm. Therefore the state change counter is used as a measure if the communication is ok (state OK and counter 0, or node not in list), completely failed (state DELAYED and counter 0) or acting randomly (counter over 1).
- Slowly decrementing state counter was chosen to keep the entries with most state changes longer on the list.
- Auto eviction algorithm aims to always keep majority of the group live or to avoid eviction. Otherwise cluster would disintegrate quickly by cross auto eviction if the connectivity between all nodes would have problems.
Suggested parameter values for testing with default EVS values:
evs.delayed_period=PT2S
evs.delayed_decay_period=PT30S
evs.auto_evict=5
ayurchen
ronin13
dirtysalt
This will be parent ticket for further packet loss related work.
Original report: https://bugs.launchpad.net/percona-xtradb-cluster/+bug/1274192
Introducing high delay and packet loss into network will make inter node communication highly unreliable in form of duplicated, lost or delayed messages. These kind of conditions will bring up some EVS related bugs, like currently open #37 and #40. EVS protocol related bugs are not in the scope of this ticket and should be reported and fixed separately.
Further work can be divided roughly in three parts, which will be outlined below.
1. Monitoring and manual eviction
Automatic node isolation or eviction is currently not possible since EVS lacks some necessary elements like proper per-node statistics collection and protocol to communicate current view of node states without running full membership protocol. Running full membership protocol in unstable network to should be avoided as it may result in oscillating membership if the network conditions are difficult enough. Therefore the first implementation of node eviction should be based on monitoring and manual eviction until better understanding about automatic eviction has been gained.
Proposed implementation:
evs.evictwsrep provider option taking list of UUIDs that should be evicted manually from cluster2. Join time health check
Joining a node with bad network connection is problematic since join operation will start EVS membership protocol round which in turn performs poorly in unstable network. To avoid starting join operation over unstable network, additional health check phase for GMCast should be deviced.
When GComm protocol stack is started, joiner should first connect to all known peers in GMCast network and exchange keep alive packets to verify that the network is ok. Joining node will start upper GComm protocol layers only after health check passes. Other nodes should not treat joining node as fully qualified member in the GMCast network until joiner sends first upper level protocol packet.
3. Automatic eviction
After enough understanding has been gained about how to properly identify the node that is causing turbulence for group communication, automatic eviction protocol can be enabled. This work will require proper per-node statistics collection on EVS level and protocol extension to communicate statistics related view to other nodes.