pqarmitage
commented
2 years ago I am confused about what is actually happening here. In the logs it shows that qg-6c2ee5e0-ad is down, whereas the ip r output appears to show that it is not (for me it shows linkdown against routes whose interface is down). Also the logs show
Sat Jan 8 12:32:51 2022: Netlink: error: No route to host(113), type=RTM_NEWROUTE(24), seq=1641645180, pid=0
Sat Jan 8 12:32:51 2022: Netlink: error: Network is unreachable(101), type=RTM_NEWROUTE(24), seq=1641645181, pid=0
Sat Jan 8 12:32:51 2022: Netlink: error: Network is down(100), type=RTM_NEWROUTE(24), seq=1641645182, pid=0which means that keepalived has not succeeding in adding the 3 virtual routes when VR_13 transitions to master, for the reasons stated.
From my testing, in order to get the Netlink: error:s above and the Error 100 (Network is down) sending gratuitous ARP messages, the link has to be administratively down (i.e. ip link set qg-6c2ee5e0-ad down).
It appears that some time between 12:32:51 and 12:33:24 interface qg-6c2ee5e0-ad was set up, and the reload then occurred after that.
What is really confusing is that if qg-6c2ee5e0-ad is set down administratively, the routes
default via x.x.244.67 dev qg-6c2ee5e0-ad
x.x.244.64/26 dev qg-6c2ee5e0-ad proto kernel scope link src x.x.244.116
x.x.244.128/25 dev qg-6c2ee5e0-ad scope linkare deleted. I therefore cannot see how the ip r output above can match with the keepalived logs.
I have tried your configuration, albeit not using openstack/neutron, and a simple load/reload of keepalived v2.2.4 does not exhibit the problem.
The routes added by keepalived v2.2.4 all have proto 18 (this is proto keepalived if you have an up-to-date iproute2 package), and you can see this in the ip r output after the reload. keepalived v1.3.9 does not add the proto 18 to the routes.
It looks to me as though the
default via x.x.244.67 dev qg-6c2ee5e0-ad
x.x.244.64/26 dev qg-6c2ee5e0-ad proto kernel scope link src x.x.244.116
x.x.244.128/25 dev qg-6c2ee5e0-ad scope linkroutes were left over when keepalived v1.3.9 terminated, and therefore the eVIP x.x.244.116/26 must also have remained configured. I am guessing that keepalived v1.3.9 was killed with SIGKILL rather than SIGTERM.
So my theory is that keepalived v1.3.9 was killed with SIGKILL, causing it to leave the ipaddresses and routes configured. keepalived v2.2.4 was then started. qg-6c2ee5e0-ad was in some state such that the old routes configured on it were not deleted, but it was down and hence the log messages, and keepalvied v2.2.4 not adding the routes. qg-6c2ee5e0-ad came up at some time between 12:32:51 and 12:33:24 and keepalived reloaded its configuration at 12:35:56, and it was then able to add the routes (which specify proto 18).
I have tried your configuration, albeit not using openstack/neutron, and a simple load/reload of keepalived v2.2.4 does not exhibit the problem; indeed the routes added by keepalived always have proto 18. I have also tested it by manually creating the routes that I believe were left over by keepalived v1.3.9 and then started keepalived v2.2.4, and I can see the duplicate routes that you have observed.
If keepalived v2.2.4 is killed with SIGKILL and then started again, at startup it removes the VIPs and routes that were left over, and then when the instance becomes master again it adds them back. The reason that the keepalived v1.3.9 routes are removed are because keepalived tries to remove routes with proto 18, and they don't exist.
While there might be a bug in keepalived v1.3.9 that caused the routes and IP addresses not to be removed when keepalived terminated (although I suspect it more likely that they were left because the VRRP keepalived process was sent SIGKILL), I don't see anything wrong with what keepalived v2.2.4 is doing. It could be argued that keepalived v2.2.4 should attempt to remove the routes without proto 18 as well, but I think the circumstances of what is happening here are so specific (i.e. keepalived is abnormally terminated, and then a new version of keepalived is run) that I think it is not worth adding the code to handle that.
If you can shed any light on the state of qg-6c2ee5d0-ad so that it appears to be down but the old routes are not deleted, then I would be very interested to understand that.
mstinsky
Describe the bug We are using keepalived in an openstack installation for the ha router implementation. After we upgraded keepalived from 1.3.9 to 2.2.4 we have problems with duplicated routes in the respective router namespaces after keepalived is reloaded by neutron.
Routes on a fresh keepalived start:
After that I can force the duplicated routes in the same way neutron is reloading keepalived.
kill -HUP ${pid}Routes after the reload of keepalived:
I am not observing the same behaviour with ipv6 routes. For ipv6 the proto 18 is just added to the existing route.
Ipv6 route before keepalived reload:
ipv6 route after keepalived reload:
To Reproduce Any steps necessary to reproduce the behaviour: n/a
Expected behavior No duplicated rules with proto 18 after a keepalived reload.
Keepalived version
Distro (please complete the following information):
Details of any containerisation or hosted service (e.g. AWS) We are running keepalived inside containers. In our case the base container is ubuntu:bionic.
Configuration file:
Notify and track scripts
System Log entries
Did keepalived coredump? n/a
Additional context I am not 100% sure if this is a keepalived issue or something related to openstack neutron working together with keepalived 2.2.4. But as this happens on a manual keepalived reload, I have the feeling that their might happen something wrong in keepalived. Any help on this is highly appreciated.