search

ClusterLabs / resource-agents

Combined repository of OCF agents from the RHCS and Linux-HA projects
GNU General Public License v2.0
493 stars 582 forks source link

IPaddr2 gratuitous ARP with CLUSTER IP contains hardware MAC instead of virtual one #1131

Open tverdoff opened 6 years ago

tverdoff commented 6 years ago

Hello community,

I have trapped into the issue with ARP cache on a router which remembers node physical MAC instead of using virtual MAC (corosync+pacemaker 1.1.15/1.1.18 cluster with cloned VIP).

Here it is:

We have cluster:

STR: 1) Start cluster and ping vip 2) on 10.0.2.12

arp -vn | grep 10.0.2.120 10.0.2.120 ether 03:00:60:0d:f0:0d C eth0

MAC is correct refers to virtual mac

3) on node1 (10.0.2.10) sudo crm node standby

sending node to standby mode

4) on 10.0.2.12

> arp -vn | grep 10.0.2.120
10.0.2.120   ether   03:00:60:0d:f0:0d  C eth0

nothing happens 10.0.2.12 still refers to virtual mac for 10.0.2.120 vip

5) again on node1 (10.0.2.10) > sudo crm node online

6) on 10.0.2.12 now we see gratuitous ARPs

sudo tcpdump -i eth0 -ennq -t \(arp\)
0a:66:34:74:ee:8d > ff:ff:ff:ff:ff:ff, ARP, length 42: Request who-has 10.0.2.120 (ff:ff:ff:ff:ff:ff) tell 10.0.2.120, length 28
0a:66:34:74:ee:8d > ff:ff:ff:ff:ff:ff, ARP, length 42: Request who-has 10.0.2.120 (ff:ff:ff:ff:ff:ff) tell 10.0.2.120, length 28
0a:66:34:74:ee:8d > ff:ff:ff:ff:ff:ff, ARP, length 42: Request who-has 10.0.2.120 (ff:ff:ff:ff:ff:ff) tell 10.0.2.120, length 28
0a:66:34:74:ee:8d > ff:ff:ff:ff:ff:ff, ARP, length 42: Request who-has 10.0.2.120 (ff:ff:ff:ff:ff:ff) tell 10.0.2.120, length 28
0a:66:34:74:ee:8d > ff:ff:ff:ff:ff:ff, ARP, length 42: Request who-has 10.0.2.120 (ff:ff:ff:ff:ff:ff) tell 10.0.2.120, length 28
ce:f7:b7:e3:3d:f3 > ff:ff:ff:ff:ff:ff, ARP, length 42: Request who-has 10.0.2.120 (ff:ff:ff:ff:ff:ff) tell 10.0.2.120, length 28
ce:f7:b7:e3:3d:f3 > ff:ff:ff:ff:ff:ff, ARP, length 42: Request who-has 10.0.2.120 (ff:ff:ff:ff:ff:ff) tell 10.0.2.120, length 28
0a:66:34:74:ee:8d > ff:ff:ff:ff:ff:ff, ARP, length 42: Request who-has 10.0.2.120 (ff:ff:ff:ff:ff:ff) tell 10.0.2.120, length 28
0a:66:34:74:ee:8d > ff:ff:ff:ff:ff:ff, ARP, length 42: Request who-has 10.0.2.120 (ff:ff:ff:ff:ff:ff) tell 10.0.2.120, length 28
ce:f7:b7:e3:3d:f3 > ff:ff:ff:ff:ff:ff, ARP, length 42: Request who-has 10.0.2.120 (ff:ff:ff:ff:ff:ff) tell 10.0.2.120, length 28
ce:f7:b7:e3:3d:f3 > ff:ff:ff:ff:ff:ff, ARP, length 42: Request who-has 10.0.2.120 (ff:ff:ff:ff:ff:ff) tell 10.0.2.120, length 28
0a:66:34:74:ee:8d > ff:ff:ff:ff:ff:ff, ARP, length 42: Request who-has 10.0.2.120 (ff:ff:ff:ff:ff:ff) tell 10.0.2.120, length 28
0a:66:34:74:ee:8d > ff:ff:ff:ff:ff:ff, ARP, length 42: Request who-has 10.0.2.120 (ff:ff:ff:ff:ff:ff) tell 10.0.2.120, length 28
ce:f7:b7:e3:3d:f3 > ff:ff:ff:ff:ff:ff, ARP, length 42: Request who-has 10.0.2.120 (ff:ff:ff:ff:ff:ff) tell 10.0.2.120, length 28
ce:f7:b7:e3:3d:f3 > ff:ff:ff:ff:ff:ff, ARP, length 42: Request who-has 10.0.2.120 (ff:ff:ff:ff:ff:ff) tell 10.0.2.120, length 28
0a:66:34:74:ee:8d > ff:ff:ff:ff:ff:ff, ARP, length 42: Request who-has 10.0.2.120 (ff:ff:ff:ff:ff:ff) tell 10.0.2.120, length 28
ce:f7:b7:e3:3d:f3 > ff:ff:ff:ff:ff:ff, ARP, length 42: Request who-has 10.0.2.120 (ff:ff:ff:ff:ff:ff) tell 10.0.2.120, length 28
ce:f7:b7:e3:3d:f3 > ff:ff:ff:ff:ff:ff, ARP, length 42: Request who-has 10.0.2.120 (ff:ff:ff:ff:ff:ff) tell 10.0.2.120, length 28
ce:f7:b7:e3:3d:f3 > ff:ff:ff:ff:ff:ff, ARP, length 42: Request who-has 10.0.2.120 (ff:ff:ff:ff:ff:ff) tell 10.0.2.120, length 28
ce:f7:b7:e3:3d:f3 > ff:ff:ff:ff:ff:ff, ARP, length 42: Request who-has 10.0.2.120 (ff:ff:ff:ff:ff:ff) tell 10.0.2.120, length 28

However nobody among cluster nodes answers request (should they?) and 10.2.0.12 gets MAC of the recently brought up node1 instead of virtual MAC

> arp -vn | grep 10.0.2.120
10.0.2.120               ether   0a:66:34:74:ee:8d   C                     eth0

Anyway gratuitous ARP request contains hardware MAC address, but not virtual.

arp_tcpdump

Since network router has ARP cache it caches the wrong mac for 4 hours in particular and a half of requests to the VIP fails then.

As a workaround we can decrease ARP cache timeout on the router, but isn't it a bug?

Could you clarify if it this behavior is expected? Can we fix it on the cluster side?

Have read mail queue https://lists.clusterlabs.org/pipermail/pacemaker/2009-October/026763.html which seems similar to me, but I don't have such an error when executing send_arp command.

on the node1

> sudo /usr/libexec/heartbeat/send_arp -i 200 -r 5 -p /var/run/resource-agents/send_arp-10.0.2.120 eth0 10.0.2.120 0300600df00d not_used not_used
ARPING 10.0.2.120 from 10.0.2.120 eth0
Sent 5 probes (5 broadcast(s))
Received 0 response(s)

on the 10.0.2.12

> sudo tcpdump -i eth0 -ennq -t \(arp\)
0a:66:34:74:ee:8d > ff:ff:ff:ff:ff:ff, ARP, length 42: Request who-has 10.0.2.120 (ff:ff:ff:ff:ff:ff) tell 10.0.2.120, length 28
0a:66:34:74:ee:8d > ff:ff:ff:ff:ff:ff, ARP, length 42: Request who-has 10.0.2.120 (ff:ff:ff:ff:ff:ff) tell 10.0.2.120, length 28
0a:66:34:74:ee:8d > ff:ff:ff:ff:ff:ff, ARP, length 42: Request who-has 10.0.2.120 (ff:ff:ff:ff:ff:ff) tell 10.0.2.120, length 28
0a:66:34:74:ee:8d > ff:ff:ff:ff:ff:ff, ARP, length 42: Request who-has 10.0.2.120 (ff:ff:ff:ff:ff:ff) tell 10.0.2.120, length 28
0a:66:34:74:ee:8d > ff:ff:ff:ff:ff:ff, ARP, length 42: Request who-has 10.0.2.120 (ff:ff:ff:ff:ff:ff) tell 10.0.2.120, length 28

Please let me know if more info is required

kskmori commented 6 years ago

Hi,

I would appreciate if you could try with v4.1.0 or later which contains #977 that includes a related fix.

IIUC, the behavior of sending a hardware MAC as gratuitous ARPs is not expected in this case and must be accidentally introduced in the past. #977 fixes it so that it will not send gratuitous ARPs when it is a cluster IP, so you should no longer see them at step 6) in your scenario and routers would continue to refer to the virtual MAC. I believe it works.

tverdoff commented 6 years ago

Hi, OK, thanks for the update, I'll let you know the result

ethaniel commented 5 years ago

Looks like updating the resource-agents solved this problem.

Here is what I had to do on my Centos 7, because there is no latest rpm available yet.

cd /tmp
git clone https://github.com/ClusterLabs/resource-agents.git
cd resource-agents/
./autogen.sh

yum -y install docbook-style-xsl glib2-devel
./configure
make
rm -fv /usr/lib/ocf/resource.d/heartbeat/.ocf-*
make install

I had to do rm -fv /usr/lib/ocf/resource.d/heartbeat/.ocf-* because otherwise make install fails for some reason (probably worth looking into).

After upgrading the resource-agents by hand, my system is not sending out ARPs anymore:

Nov 18 19:43:33 haproxy-02 IPaddr2(cluster_ip:0)[1796657]: INFO: /usr/libexec/heartbeat/send_arp  -i 200 -r 5 -p /var/run/resource-agents/send_arp-10.0.0.80 eth1 10.0.0.80 35fc518e5745 not_used not_used
Nov 18 19:43:33 haproxy-02 IPaddr2(cluster_ip:0)[1796657]: INFO: send_arp.linux: Gratuitous ARPs are not sent in the Cluster IP configuration
buczo23 commented 5 years ago

Any updates?