search

openinfrastructure / terraform-google-multinic

Connect two VPC networks with an auto-healing, auto-scaling group of IP router instances.
Apache License 2.0
10 stars 5 forks source link

Make systemctl restart google-network-daemon work #2

Closed jeffmccune closed 5 years ago

jeffmccune commented 5 years ago

Summary

Consider a vpc-link instance which has completed startup and is forwarding packets correctly. When the google-network-daemon service is restarted, the instance no longer forwards packets correctly.

When a client (10.19.16.45) on the local VPC attached to eth1, which pings 10.0.0.6 in the shared VPC, the ICMP reply generates a huge number of packets which are "stuck" in eth0:

On the client (10.19.16.45) instance:

ping 10.0.0.6 -c 1

On the IP router instance, note the ICMP echo reply getting "stuck" in of eth0, when it should be forwarded out eth1.

sudo tcpdump -i eth0 host 10.0.0.6 -n
00:22:27.876266 IP 10.19.16.45 > 10.0.0.6: ICMP echo request, id 25139, seq 1, length 64
00:22:27.881121 IP 10.0.0.6 > 10.19.16.45: ICMP echo reply, id 25139, seq 1, length 64
00:22:27.881126 IP 10.0.0.6 > 10.19.16.45: ICMP echo reply, id 25139, seq 1, length 64
00:22:27.881225 IP 10.0.0.6 > 10.19.16.45: ICMP echo reply, id 25139, seq 1, length 64
00:22:27.881230 IP 10.0.0.6 > 10.19.16.45: ICMP echo reply, id 25139, seq 1, length 64
<huge number of same id packets>
00:22:27.882245 IP 10.0.0.6 > 10.19.16.45: ICMP echo reply, id 25139, seq 1, length 64
00:22:27.882301 IP 10.0.0.6 > 10.19.16.45: ICMP echo reply, id 25139, seq 1, length 64
00:22:27.882325 IP 10.0.17.55 > 10.0.0.6: ICMP time exceeded in-transit, length 92

Root cause

The default via 10.0.3.1 dev eth1 route should exist in table rt1, but does not after dhclient eth1 completes, run by google-network-daemon:

Correct table

# ip route show table rt1
default via 10.0.3.1 dev eth1
10.0.3.0/24 dev eth1 scope link src 10.0.3.55

Incorrect table

The following is the table rt1 after dhclient operates against eth1. Note the missing default route, which causes the packet to go back out eth0.

# ip route show table rt1
10.0.3.0/24 dev eth1 scope link src 10.0.3.55

Next steps

  1. Configure ip route add default via "${gateway}" dev eth1 table rt1 using a dhclient exit hook.
  2. Do not configure route table setup and rules via dhclient exit hooks, otherwise they stack up. Ensure ip rules are only configured once on boot.

/etc/hosts to easily track cases:

10.0.0.6    server-east
10.0.17.55  router-east # eth0
10.0.3.55   router-west # eth1
10.19.16.45 client-west

Verification

jeffmccune commented 5 years ago

Note, /usr/sbin/dhclient-script does not set $routers when classless static routes are provided by the DHCP server, which is the case for secondary network interfaces in GCP:

402         if [ -n "${new_classless_static_routes}" ] ||
403            [ -n "${new_static_routes}" ]; then
404             if [ -n "${new_classless_static_routes}" ]; then
405                 IFS=', |' static_routes=(${new_classless_static_routes})
406                 # If the DHCP server returns both a Classless Static Routes option and
407                 # a Router option, the DHCP client MUST ignore the Router option. (RFC3442)
408                 new_routers=""
409             else
410                 IFS=', |' static_routes=(${new_static_routes})
411             fi

This means we can't rely on dhclient to provide the gateway IP address for non-primary interfaces.

jeffmccune commented 5 years ago

Fixed in 0.5.0